Use Azure Private Link with Confluent Cloud

Azure Private Link allows for one-way secure connection access from your VNet to Confluent Cloud with added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity of setup.

The following diagram summarizes the Azure Private Link architecture between the VNet or subscription and the Confluent Cloud cluster.

To set up to use Azure Private Link with Confluent Cloud:

1. Identify a Confluent Cloud network you want to use, or set up a new Confluent Cloud network.
2. Add a Private Link access in Confluent Cloud.
3. Provision Private Link endpoints in Azure.
4. Set up DNS records in Azure.
5. Validate connectivity.
6. Troubleshoot broker connectivity issues if necessary.

The following tutorial video walks you through the process of configuring a private link. It also covers common mistakes and gotchas when setting up a private link with Confluent Cloud. Even though it uses AWS PrivateLink as an example, you will still find it relevant and useful for setting up a Private Link in Azure.

Requirements and considerations

Review the following requirements and considerations before you set up a PrivateLink in Azure with Confluent Cloud:

• Have a Confluent Cloud network of type Private Link in Azure. If a network does not exist, see Create Confluent Cloud Network on Azure.

• To use an Azure Private Link service with Confluent Cloud, your VNet must allow outbound Internet connections for Confluent Cloud Schema Registry, ksqlDB, and Confluent CLI to work.

  • DNS requests to public authority traversing to private DNS zone is required.
  • Confluent Cloud Schema Registry is accessible over the Internet.
  • Provisioning new ksqlDB instances requires Internet access. After ksqlDB instances are up and running, they are fully accessible over Azure Private Link connections.
  • Confluent CLI requires Internet access to authenticate with the Confluent Cloud control plane.

• Confluent Cloud Console components, such as topic management, need additional configuration to function as they use cluster endpoints. To use all features of the Confluent Cloud Console with Azure Private Link, see Use Confluent Cloud with Private Networking.

• Cross-region Azure Private Link is not supported through the Confluent Cloud Console.

  Contact Confluent Support to see if your regions are supported and to request configuration.

• Azure Private Link is only available for use with Dedicated clusters.

• Existing Confluent Cloud clusters cannot be converted to use Azure Private Link.

• Confluent Cloud selects the availability zones for the Confluent Cloud cluster and Azure Private Link service. You cannot select availability zones.

Connectors

Fully-managed Confluent Cloud connectors can connect to data sources or sinks using a public IP address.

Fully-managed Confluent Cloud connectors can use Egress Access Endpoints to connect to the sources or sinks in the customer network with private IP addresses. For information about configuring an Egress Access Point, see Use Azure Egress Access Points for Dedicated Clusters on Confluent Cloud.

Add a Private Link access in Confluent Cloud

Azure Private Link consists of two parts: the Private Link Service exposed by Confluent Cloud and private endpoints that you configure within your Azure subscription.

To make an Azure Private Link connection to a cluster in Confluent Cloud, first, register your Azure subscription with the Confluent Cloud network for automatic approval of private endpoint connections. This adds a Private Link access in Confluent Cloud. If required, you can register multiple subscriptions.

1. Open the Confluent Cloud Console, in the Network Management tab, click the Confluent Cloud network you want to add the connection to.
2. Click + Private Link Access.
3. Specify the following values in the fields:
   • Name: The name for this private link access.
   • Azure subscription ID: The Azure subscription ID for the account containing the VNets you want to make the Azure Private Link connection from. The Azure subscription number can be found on your Azure subscription page of the Azure portal.
4. Click Add.

When your Azure Private Link connection status transitions to Ready in the Confluent Cloud Console, configure the private endpoints in your VNet before you can connect to the cluster.

Provision Private Link endpoints in Azure

When the Private Link access becomes ready in the Confluent Cloud Console, configure private endpoints in your VNet from the Azure portal to make the Azure Private Link connection to your Confluent Cloud cluster. endpoints. This configuration automates the manual steps described below.

For Confluent Cloud single availability zone clusters, create a single private endpoint to the Confluent Cloud Service Alias for the Kafka cluster zone.

For Confluent Cloud multi-availability zone clusters, create three private endpoints, one endpoint to each of the Confluent Cloud zonal Service Aliases.

To set up the VNet endpoint for Azure Private Link in your Azure account:

1. In the Confluent Cloud Console, gather the following information in Cluster Overview.

   • In Cluster Settings:

     • Bootstrap server endpoint in the Endpoints section

     • Zones in the Cloud details section

       The availability zones of the Kafka cluster.

   • In Networking > Details:

     • DNS domain

     • DNS subdomain

     • Service aliases

       For single availability zone clusters, you only need the Service Alias of the Kafka cluster zone (Zones) that you retrieved in Cluster Settings > Cloud details. above.

2. In the Azure Private Link Center, click Create private endpoint.

   Specify the required fields as described in Create a private endpoint.

   • Selected Subscription must be the name of the subscription ID that you registered with Confluent Cloud in Add a Private Link access in Confluent Cloud. You can check it on the Portal Subscriptions page.
   • Name: Specify an instance name.
   • Network Interface Name: Use the default given.
   • Region: Select the same region as the Confluent Cloud Private Link access.

3. Click Next: Resource.

   • Connection method: Select the Connect to an Azure resource by resource ID or alias.

   • Resource ID or alias: Specify the Confluent Cloud Service Alias. and click Next.

     You can find the Confluent Cloud Service Aliases on the Confluent Cloud Console, in the Networking tab under Cluster settings.

     For a Confluent Cloud single availability zone cluster, use the Service Alias for the Kafka cluster availability zone.

     
     
     
     
     
     

     For a Confluent Cloud multi-availability zone cluster, use the Confluent Cloud zonal Service Alias for each private endpoint. Note that you need three private endpoints.

4. Click Next: Virtual Network.

   • Virtual network: The virtual network where the private endpoint is to be created.

   • Subnet: The subnet where the private endpoint is to be created.

     You must select the subnets in the same region and availability zone as your Confluent Cloud network from the Networking tab in the Confluent Cloud Console.

5. Click Review + create. Review the details and click Create to create the private endpoint.

6. Wait for the Azure deployment to complete, go to the private endpoint resource, and verify the private endpoint connection status is Approved.

Set up DNS records in Azure

You must update your DNS records to ensure connectivity passes through Azure Private Link in the supported pattern. Any DNS provider that can ensure DNS is routed as follows is acceptable. Azure Private DNS Zone (used in this example) is one option.

DNS resolution options

For Azure Private Link Confluent Cloud networks, you can use the public or private DNS resolution:

• The private DNS resolution is the recommended option and guarantees fully private DNS resolution.
• The public DNS resolution is useful when you want to ensure that Confluent deployments are homogenous and conform to DNS configurations for your networks.

DNS resolution is selected when you create a Confluent Cloud network, and it cannot be modified after creating the Confluent Cloud network. See Create a Confluent Cloud network.

Public DNS resolution

The public (also known as chased private in Confluent Cloud) DNS resolution is used for the bootstrap server and broker hostnames of a Confluent Cloud cluster that is using Azure Private Link. When the public resolution is used, the clusters in this network require both public and private DNS to resolve cluster endpoints.

Only the Confluent Global DNS Resolver (GLB) endpoints are advertised.

The public DNS resolution performs the following two-step process:

1. The Confluent Cloud Global DNS Resolver removes the glb subdomain and returns a CNAME for your bootstrap and broker hostnames.

   Example: $lkc-id-$nid.$region.$cloud.glb.confluent.cloud

   CNAME returned: $lkc-id.$nid.$region.$cloud.confluent.cloud

2. The CNAME resolves to your VNet private endpoints based on the Private DNS Zone configuration.

3. Private DNS resolution

When the private DNS resolution is used, the clusters in this network only require private DNS to resolve cluster endpoints. Only non-GLB endpoints are advertised.

Create a DNS zone and DNS records

DNS entries need to be created for Private Link irrespective of the DNS resolution option you selected when creating the Confluent Cloud network. These DNS records map the Confluent Cloud DNS names to the Azure private endpoint addresses.

To update DNS resolution using Azure Private DNS Zone:

1. Browse to the Azure portal.

2. Create a Private DNS zone for the Private Link.

   1. Search for the Private DNS zone resource in the Azure portal.

   2. Click + Create.

   3. Specify the required fields as described in Create a DNS zone.

      • Name: Use the Confluent Cloud DNS domain value from the Networking under Cluster Overview in the Confluent Cloud Console.

        Note that there is no glb in the DNS Domain name.

        If the Confluent Cloud DNS Domain name includes the logical cluster ID which starts with lkc-, omit the logical cluster id when specifying it as the Private DNS Zone name. For example, the DNS Domain name shown as lkc-123abc-4kgzg.centralus.azure.confluent.cloud in Confluent Cloud should be converted to 4kgzg.centralus.azure.confluent.cloud to be used as the Private DNS Zone name.

   4. Click Review + create.

   5. Wait for the Azure deployment to complete.

3. (Optional) To identify the correct mapping of DNS zone records to zonal endpoints for Confluent Cloud, you can run the DNS helper script from your VM instance within the VNet.

   You specify the Azure resource group name, and one or more private endpoints as arguments. You can find these values form the networking detail page in the Confluent Cloud Console.

   The output contains the Azure domain names, the record types, and the DNS values that you can input when you create DNS records. For example:

   ./dns-endpoints.sh my-resource-group private-endpoint-1 private-endpoint-2 private-endpoint-3
   
   *              A    10.0.0.2 10.0.0.3 10.0.0.4
   *.az2          A    10.0.0.2
   *.az1          A    10.0.0.3
   *.az3          A    10.0.0.4
   

4. Add DNS records.

   Create required DNS records as described in Manage DNS records and record sets by using the Azure portal.

   1. Go to the Private DNS Zone resource as created above.

   2. Click + Record set.

   3. Create the record set for bootstrap endpoint resolution.

      The bootstrap DNS record should contain three zonal endpoints IPs to resolves to all three zonal endpoints that you created in Provision Private Link endpoints in Azure.

      • Name: *
      • Type: A
      • TTL: 1
      • TTL unit: Minute
      • IP address: The IP addresses of the Azure private endpoints.

      The IP address of an Azure private endpoint can be found under its associated Network interface.

      

   4. Create three zonal DNS records, one zonal DNS record per availability zone of the Confluent Cloud network.

      The IP address of the private endpoint can be found under its associated network interface.

      • Name: *.az1
      • Type: A
      • TTL: 1
      • TTL unit: Minute
      • IP address: IP address of the az1 private endpoint as created above.
      • Name: *.az2
      • Type: A
      • TTL: 1
      • TTL unit: Minute
      • IP address: IP address of the az2 private endpoint as created above.
      • Name: *.az3
      • Type: A
      • TTL: 1
      • TTL unit: Minute
      • IP address: IP address of the az3 private endpoint as created above.

   Note

   In Confluent Cloud with private linking, Kafka broker names you retrieve from the metadata are not static. Do not hardcode the broker names in DNS records.

   The following are example DNS record sets.

   

5. Attach the Private DNS Zone to the VNets where clients or applications are present.

   1. Go to the Private DNS Zone resource and click Virtual network links under Settings.
   2. Click Add.
   3. Fill in Link name, Subscription, and Virtual network.
   4. Click OK.

Creating Network Interfaces

 

Creating Network Interfaces

A network interface enables an Azure VM to communicate with the Internet, Azure, and other on-premises resources. A VM can have one or more network interfaces.

For Aruba Virtual Gateway deployment, Aruba recommends that you create a separate network interface for each traffic type and map these to subnets in the following order:

• NIC1—Management traffic
• NIC2—Internet traffic– Requires a Public IP for connectivity to Aruba Central
• NIC3—VPN traffic
• NIC4—LAN traffic

1. To create network interfaces follow these steps:
   1. In the Azure portal, click Home > +Create a Resource .
   2. On the Home page, enter network interface in the search box to access the Network Interface page.
   3. Click Create.
   4. In the Create network interface window, configure the following parameters:
   • Name—Enter a name for the interface; for example, NIC1.
   • Virtual network—Select the VNET to which you want to add the interface.
   • Subnet—Select the subnet that you want assign to the interface.
   • Private IP address assignment— Based on your configuration requirements, select either Dynamic or Static addressing.
   • Network security group—Select the network security group to which you want to attach this interface.
   • Subscription—Select the subscription.
   • Resource Group—Select the resource group to which you want to map the interface.
   • Location—Select the geographic location of your VNET.
   5. Click Create.
   6. Repeat these steps to create additional network interfaces, for NIC2, NIC3, and NIC4.

   Figure 1  Creating Network Interfaces

   

2. To enable IP forwarding for NIC2, NIC3, and NIC4, follow these steps:
   1. In the Azure portal, go to Home > Resource groups.
   2. Select the resource group associated to your VNET.
   3. From the list of network interfaces, select the NIC.
   4. Under Settings, select IP configurations.
   5. Configure the following parameters:

   IP forwarding settings

   IP forwarding— Set to Enabled.

   Figure 2  Enabling IP forwarding

   

3. To add Public IP to iNeT, follow these steps:
   1. Under Settings, select IP configurations.
   2. Set IP forwarding to Enabled.
   3. Click Save.
   4. Click the listed ipconfig1.
   5. In the new window, set the Public IP address as Enabled.
   6. Click Save.

 

Add network interfaces to an Azure Virtual Machine

Supported Versions

Conductor and Azure Cloud Airwall Gateways v3.0 and later

Required Role

• System or network administrators
• Permissions to edit the Azure Cloud Airwall Gateways. You need to be a manager of at least one overlay that these Airwall Gateways are in.

Supported on these Airwall Edge Services

Airwall Gateway 300v only.

Before you Begin

Before you add an interface, consider:

• Cost – An Azure virtual machine (VM) that supports multiple interfaces increases the cost, so weigh the costs and benefits of adding more network interfaces versus deploying multiple 300v Airwall Gateways.

• Machine type – If you are using the default Azure VM, Standard A2 v2, you must upgrade the machine type to one that supports the number of NICs you want. A list of available NICs per machine type is available in the Azure documentation. Select a machine series from the list (for example, the Av2-series) and scroll to the right in the table to find the maximum number of NICs.

• Update your Conductor and Airwall Gateway 300v to v3.0 – v3.0 is required to manage the route injection for devices on the additional network interfaces.

Add a Network Interface (NIC) to an Azure 300v deployment

These instructions may change. Please refer to the Azure documentation for updated information.

1. Go to the Azure VM that is hosting your 300v Airwall Gateway.
2. Stop the Azure VM.
   
   

   

   
   
3. Go to Settings > Size, and select a new machine type. This screenshot shows upgrading to a Standard A4 v2 that allows up to 4 NICs.
   
   

   

   
   
4. In the 300v’s Resource Group, create a new Network Interface object (NIC).
   
   

   

   
   
5. Under the Azure VM’s SettingsNetworking, select Attach network interface and under Attach existing network interface, select the network interface you created.
   
   

   

   
   
6. Restart the Azure VM.
   
   

   

   
   

Once the Azure VM reconnects to the Conductor, the 300v updates its port information with a third network interface with the overlay gateway IP already configured:

 

Manage VM network interfaces for on-premises disaster recovery to Azure

• 
  

A virtual machine (VM) in Azure must have at least one network interface attached to it. It can have as many network interfaces attached to it as the VM size supports.

By default, the first network interface attached to an Azure virtual machine is defined as the primary network interface. All other network interfaces in the virtual machine are secondary network interfaces. Also by default, all outbound traffic from the virtual machine is sent out the IP address that's assigned to the primary IP configuration of the primary network interface.

In an on-premises environment, virtual machines or servers can have multiple network interfaces for different networks within the environment. Different networks are typically used for performing specific operations such as upgrades, maintenance, and internet access. When you're migrating or failing over to Azure from an on-premises environment, keep in mind that network interfaces in the same virtual machine must all be connected to the same virtual network.

By default, Azure Site Recovery creates as many network interfaces on an Azure virtual machine as are connected to the on-premises server. You can avoid creating redundant network interfaces during migration or failover by editing the network interface settings under the settings for the replicated virtual machine.

Select the target network

For VMware and physical machines, and for Hyper-V (without System Center Virtual Machine Manager) virtual machines, you can specify the target virtual network for individual virtual machines. For Hyper-V virtual machines managed with Virtual Machine Manager, use network mapping to map VM networks on a source Virtual Machine Manager server and target Azure networks.

1. Under Replicated items in a Recovery Services vault, select any replicated item to access the settings for that replicated item.

2. Select the Compute and Network tab to access the network settings for the replicated item.

3. Under Network properties, choose a virtual network from the list of available network interfaces.

   

Modifying the target network affects all network interfaces for that specific virtual machine.

For Virtual Machine Manager clouds, modifying network mapping affects all virtual machines and their network interfaces.

Select the target interface type

Under the Network interfaces section of the Compute and Network pane, you can view and edit network interface settings. You can also specify the target network interface type.

• A Primary network interface is required for failover.
• All other selected network interfaces, if any, are Secondary network interfaces.
• Select Do not use to exclude a network interface from creation at failover.

By default, when you're enabling replication, Site Recovery selects all detected network interfaces on the on-premises server. It marks one as Primary and all others as Secondary. Any subsequent interfaces added on the on-premises server are marked Do not use by default. When you're adding more network interfaces, ensure that the correct Azure virtual machine target size is selected to accommodate all required network interfaces.

Modify network interface settings

You can modify the subnet and IP address for a replicated item's network interfaces. If an IP address is not specified, Site Recovery will assign the next available IP address from the subnet to the network interface at failover.

1. Select any available network interface to open the network interface settings.

2. Choose the desired subnet from the list of available subnets.

3. Enter the desired IP address (as required).

   

4. Select OK to finish editing and return to the Compute and Network pane.

5. Repeat steps 1-4 for other network interfaces.

6. Select Save to save all changes.