AZ Asia-Pacific - Zscaler Help Centre
All Categories Zscaler Private Access ZPA App Connector : Ensuring Best Performance

ZPA App Connector : Ensuring Best Performance

ZPA App Connectors that are deployed in the customer environments provide connectivity to client applications where they are provided as an OVA for installation in VMWare environments, or as an AMI for deployment in AWS – in both cases it is a CentOS 7 image which has been hardened by removing unnecessary services and listeners. The App Connector is also available as an RPM for installation on Enterprise Linux platforms.

In all cases, once deployed, the customer is responsible for operating system patching and maintenance. Performing regular maintenance, log rotation, operating system monitoring is important. Similarly, the App Connector OVA/AMI is updated periodically with a patched Operating System and latest release of software – the App Connectors could be re-deployed using standard DevSecOps processes to ensure they are always at a base level.

Whilst the base App Connector may be deployed, or the RPM installed, several steps should be considered to ensure the Zscaler processes run appropriately and can make best use of the resources.

This article describes the changes that may be necessary.

Changes to Operating System

  1. Network Interface

    DHCP may be used for connectors, especially in IaaS environments. However, best practice would be to use a static IP address. If IPv6 is not available on the network, then disable it on the network interface also.

    Configuring appropriate DNS servers that are in the same location, or geographically close to the App Connector will ensure DNS response time is optimized. Whilst ZPA will cache responses, DNS performance is critical to operation.

    Sample configuration for reference:

    #Configure Network Interface

    cat > /etc/sysconfig/network-scripts/ifcfg-eth0 <<-EOT

    DEVICE=eth0

    BOOTPROTO=none

    ONBOOT=yes

    TYPE=Ethernet

    USERCTL=yes

    IPADDR=<IPADDR>

    PREFIX=</24>

    GATEWAY=<IPGATEWAY>

    DNS1=<DNS SERVER 1>

    DNS2=<DNS SERVER 2>

    DEFROUTE=yes

    IPV6INIT=no

    Once the network settings have been applied, it is important to restart the network processes to re-read the configuration. Zscaler Private Access processes should be stopped before network process restart to ensure it reads the changes. Since this will interrupt network, it is preferable to perform this commands from the VM Console.

    Sample commands for reference:

    #Restart Network Interfaces

    sudo systemctl stop zpa-connector

    sudo systemctl restart network

    (make sure the network settings work)

    sudo systemctl start zpa-connector

  2. IPV6

    IPv6 is enabled by default, however this may not be needed in the network where the App Connector is deployed in. If the App Connector does not have an IPv6 address, it will not be able to connect to IPv6 Applications.

    When a client application is requested, the App Connector would make a DNS A record lookup AND a DNS AAAA record lookup for resolution. If the DNS server is not capable of responding to the AAAA record, this could add delay in resolution. Best practice is to disable IPv6 if it is not available on the DNS server or in the network

    Sample commands for reference:

    #Disable IPv6 Entirely

    sysctl -w net.ipv6.conf.all.disable_ipv6=1

    sysctl -w net.ipv6.conf.default.disable_ipv6=1

    #Persist Change Across Reboot

    cat >> /etc/sysctl.d/99-sysctl.conf <<-EOT

    net.ipv6.conf.all.disable_ipv6 = 1

    net.ipv6.conf.default.disable_ipv6 = 1

    EOT

  3. DNS

    DNS is important to all functions across Zscaler Private Access. The configuration changes on the

    network adaptor ensure an IPv6 address not initialized (if appropriate) and IPv6 settings ensure AAAA records are not requested (if appropriate).

    Linux handles /etc/resolv.conf as an ordered preference list, using secondary entries only as backups. The App Connector processes handles /etc/resolv.conf as a round-robin pool, load balancing between all entries.

    It is important to ensure the DNS servers are used correctly. DNS Servers can get overloaded, especially since Zscaler Private Access will generate a large number of queries based on the users the connector is serving. Similarly, it is important to ensure the DNS servers are functioning – since ZPA rotates between the DNS servers in /etc/resolv.conf, no DNS server should be offline. ZPA re-reads from /etc/resolv.conf the DNS servers every 5 minutes to ensure consistency.

    The timeout values and retries could be tailored to the specific environment. Rotating through DNS servers will ensure a single DNS server does not take all the requests, and having appropriate timeout (how long to wait for a response) and retries (how many times to ask the same DNS server for a request) will ensure a single DNS server is not saturated.

    Secondarily, ZPA uses DNS to resolve the Zscaler Private Access Service Edge (public or private) based on the DNS response. For Public DNS resolution, it is important that DNS servers return the same response for Zscaler entries – co2br.prod.zpath.net for example.

    Sample commands for reference:

    #Rotate through DNS server failures

    echo "options rotate timeout:1 retries:1" >> /etc/resolv.conf

  4. Increase Ephemeral Ports

    The ZPA App Connector’s IP address is used as the source for initiated connections to applications. Any connection made will use a TCP/UDP source port, and after use the port will be placed into a TIME_WAIT state before it can be re-used. The default port range is 32768-60999 (28231 available ports each for UDP and TCP). Increasing this port range will ensure the App Connector does not run out of source ports – this is especially important for Active Directory which will generate a lot of traffic for CLDAP and LDAP during AD Site discovery.

    Sample commands for reference:

    #Increase Port Range

    sysctl -w net.ipv4.ip_local_port_range="1024 65000"

    #Persist Change Across Reboot

    cat >> /etc/sysctl.d/99-sysctl.conf <<-EOT

    net.ipv4.ip_local_port_range = 1024 65000 EOT

  5. Zscaler Debug Parameters

    Occasionally, Zscaler TAC Support / Engineering Team may ask for enhanced debug statistics from a connector during a troubleshooting session. It’s important to ensure after a debug session, that these statistics are disabled. Whilst leaving them enabled should not cause issue, it’s best practice to revert to the base configuration. These can only be changed when the Zscaler processes are running.

    Sample commands for reference:

    #Update debug flags

    curl http://localhost:9000/debug/fohh?value=0

    curl http://localhost:9000/debug/wally?value=0

    curl http://localhost:9000/debug/zpath_lib?value=0

    curl http://localhost:9000/debug/zpn?value=0

    curl http://localhost:9000/debug/assistant?value=0

    curl http://localhost:9000/debug/zhealth?value=0

  6. Log Rotation

    ZPA App Connector will write logs to /var/log/messages via journalctl. It is important on any system to ensure the log files do not consume the disk partition. Having appropriate log rotation in place will take care of this, and compress older log files. This update should be customized based on your standard log rotation, however for the base AMI/OVA, this configuration will ensure the logs are rotated daily, store 7 days’ worth of logs, and compress old log files as they are rotated.

    Sample commands for reference:

    #Update log rotation

    cat > /etc/logrotate.conf <<-EOT

    # rotate log files daily

    daily

    # keep 7 days worth of backlogs

    rotate 7

    # create new (empty) log files after rotating old ones

    create

    # use date as a suffix of the rotated file

    dateext

    # uncomment this if you want your log files compressed

    compress

    # RPM packages drop log rotation information into this directory

    include /etc/logrotate.d

    # no packages own wtmp and btmp -- we'll rotate them here

    /var/log/wtmp {

    monthly

    create 0664 root utmp

    minsize 1M

    rotate 1

    }

    /var/log/btmp {

    missingok

    monthly

    create 0600 root utmp

    rotate 1

    }

    # system-specific logs may be also be configured here.

    EOT