ZPA App Connector : Ensuring Best Performance AZ Asia-Pacific

ZPA App Connectors that are deployed in the customer environments provide connectivity to client applications where they are provided as an OVA for installation in VMWare environments, or as an AMI for deployment in AWS – in both cases it is a CentOS 7 image which has been hardened by removing unnecessary services and listeners. The App Connector is also available as an RPM for installation on Enterprise Linux platforms.

In all cases, once deployed, the customer is responsible for operating system patching and maintenance. Performing regular maintenance, log rotation, operating system monitoring is important. Similarly, the App Connector OVA/AMI is updated periodically with a patched Operating System and latest release of software – the App Connectors could be re-deployed using standard DevSecOps processes to ensure they are always at a base level.

Whilst the base App Connector may be deployed, or the RPM installed, several steps should be considered to ensure the Zscaler processes run appropriately and can make best use of the resources.

This article describes the changes that may be necessary.

Changes to Operating System

Network Interface
DHCP may be used for connectors, especially in IaaS environments. However, best practice would be to use a static IP address. If IPv6 is not available on the network, then disable it on the network interface also.
Configuring appropriate DNS servers that are in the same location, or geographically close to the App Connector will ensure DNS response time is optimized. Whilst ZPA will cache responses, DNS performance is critical to operation.
Sample configuration for reference:
#Configure Network Interface
cat > /etc/sysconfig/network-scripts/ifcfg-eth0 <<-EOT
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
IPADDR=<IPADDR>
PREFIX=</24>
GATEWAY=<IPGATEWAY>
DNS1=<DNS SERVER 1>
DNS2=<DNS SERVER 2>
DEFROUTE=yes
IPV6INIT=no
Once the network settings have been applied, it is important to restart the network processes to re-read the configuration. Zscaler Private Access processes should be stopped before network process restart to ensure it reads the changes. Since this will interrupt network, it is preferable to perform this commands from the VM Console.
Sample commands for reference:
#Restart Network Interfaces
sudo systemctl stop zpa-connector
sudo systemctl restart network
(make sure the network settings work)
sudo systemctl start zpa-connector
IPV6
IPv6 is enabled by default, however this may not be needed in the network where the App Connector is deployed in. If the App Connector does not have an IPv6 address, it will not be able to connect to IPv6 Applications.
When a client application is requested, the App Connector would make a DNS A record lookup AND a DNS AAAA record lookup for resolution. If the DNS server is not capable of responding to the AAAA record, this could add delay in resolution. Best practice is to disable IPv6 if it is not available on the DNS server or in the network
Sample commands for reference:
#Disable IPv6 Entirely
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
#Persist Change Across Reboot
cat >> /etc/sysctl.d/99-sysctl.conf <<-EOT
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
EOT
DNS
DNS is important to all functions across Zscaler Private Access. The configuration changes on the
network adaptor ensure an IPv6 address not initialized (if appropriate) and IPv6 settings ensure AAAA records are not requested (if appropriate).
Linux handles /etc/resolv.conf as an ordered preference list, using secondary entries only as backups. The App Connector processes handles /etc/resolv.conf as a round-robin pool, load balancing between all entries.
It is important to ensure the DNS servers are used correctly. DNS Servers can get overloaded, especially since Zscaler Private Access will generate a large number of queries based on the users the connector is serving. Similarly, it is important to ensure the DNS servers are functioning – since ZPA rotates between the DNS servers in /etc/resolv.conf, no DNS server should be offline. ZPA re-reads from /etc/resolv.conf the DNS servers every 5 minutes to ensure consistency.
The timeout values and retries could be tailored to the specific environment. Rotating through DNS servers will ensure a single DNS server does not take all the requests, and having appropriate timeout (how long to wait for a response) and retries (how many times to ask the same DNS server for a request) will ensure a single DNS server is not saturated.
Secondarily, ZPA uses DNS to resolve the Zscaler Private Access Service Edge (public or private) based on the DNS response. For Public DNS resolution, it is important that DNS servers return the same response for Zscaler entries – co2br.prod.zpath.net for example.
Sample commands for reference:
#Rotate through DNS server failures
echo "options rotate timeout:1 retries:1" >> /etc/resolv.conf
Increase Ephemeral Ports
The ZPA App Connector’s IP address is used as the source for initiated connections to applications. Any connection made will use a TCP/UDP source port, and after use the port will be placed into a TIME_WAIT state before it can be re-used. The default port range is 32768-60999 (28231 available ports each for UDP and TCP). Increasing this port range will ensure the App Connector does not run out of source ports – this is especially important for Active Directory which will generate a lot of traffic for CLDAP and LDAP during AD Site discovery.
Sample commands for reference:
#Increase Port Range
sysctl -w net.ipv4.ip_local_port_range="1024 65000"
#Persist Change Across Reboot
cat >> /etc/sysctl.d/99-sysctl.conf <<-EOT
net.ipv4.ip_local_port_range = 1024 65000 EOT
Zscaler Debug Parameters
Occasionally, Zscaler TAC Support / Engineering Team may ask for enhanced debug statistics from a connector during a troubleshooting session. It’s important to ensure after a debug session, that these statistics are disabled. Whilst leaving them enabled should not cause issue, it’s best practice to revert to the base configuration. These can only be changed when the Zscaler processes are running.
Sample commands for reference:
#Update debug flags
curl http://localhost:9000/debug/fohh?value=0
curl http://localhost:9000/debug/wally?value=0
curl http://localhost:9000/debug/zpath_lib?value=0
curl http://localhost:9000/debug/zpn?value=0
curl http://localhost:9000/debug/assistant?value=0
curl http://localhost:9000/debug/zhealth?value=0
Log Rotation
ZPA App Connector will write logs to /var/log/messages via journalctl. It is important on any system to ensure the log files do not consume the disk partition. Having appropriate log rotation in place will take care of this, and compress older log files. This update should be customized based on your standard log rotation, however for the base AMI/OVA, this configuration will ensure the logs are rotated daily, store 7 days’ worth of logs, and compress old log files as they are rotated.
Sample commands for reference:
#Update log rotation
cat > /etc/logrotate.conf <<-EOT
# rotate log files daily
daily
# keep 7 days worth of backlogs
rotate 7
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
dateext
# uncomment this if you want your log files compressed
compress
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0600 root utmp
rotate 1
}
# system-specific logs may be also be configured here.
EOT