PrismCentral and CMSP

Question 1

What service logs do you check for AI ops/Capcity Planning/XFit? What port does it listen on

Answer 1

neuron_server.log, listens on port 2602

Question 2

When an user gets logged out immediately after logging in to PC what could be the problem

Answer 2

User might be part of more than 100 groups and iam token cannot contain more than 100 groups

Question 3

Explain dns resolution flow in Prism Central VM that has CMSP enabled for iam-proxy.ntnx-base

Answer 3

nslookup -> 127.0.0.1 (dnsmasq in PC) -> /etc/dnsmasq.conf (fail) -> External DNS fail -> Append search list to query -> dnsmaq (PC) -> master PC VM port 31000 -> resolved

Question 4

What is the eth1 ip in CMSP cluster with private network and CMSP cluster with routable ip address

Answer 4

CMSP with private ip - eth1 is 192.168.5.x network
CMSP with routable ip - eth1 is external ip address provided during enablement

Question 5

What logs contain cmsp upgrade logs

Answer 5

data/logs/cmsp_upgrade.out

Question 6

What can happen when CMSP upgrade fails

Answer 6

kubectl will not work, mspctl will report “no route to host”
cmsp_post_upgrade.py script is available to re-trigger the upgrade

Question 7

MSP upgrade fails with ssh handshake issue what could be the caused

Answer 7

msp_entity IDF object might have disable_lockdown set to true which will cause the upgrade to fail.

Question 8

Where are the post upgrade scripts for CMSP are present in Prism Central

Answer 8

Many scripts are located in /var/nutanix directory

Question 9

Describe the bootup sequence of PC and how eth1 interface is created in cmsp

Answer 9

multi-user.target -> nutanix-rc-local.service -> /usr/local/nutanix/start.d/rc.local -> start mount_disks script -> start /usr/local/nutanix/bin/cmsp_init_config.sh -> starts persist-vxlan-config.sh -> creates vxlan eth1 interface -> fixes resolver using fixup_resolv_conf.sh -> starts genesis

Question 10

how do you trigger a ncc equivalent health check in msp

Answer 10

mspctl debug run

Question 11

What does lb-controller-deployment-0 pod is used for in kube-system namespace

Answer 11

This pod monitors all services of type loadbalancer in k8s cluster, it then configures the envoy loadbalancer with the service ip and port so envoy can loadbalance the connections coming into the ports to backend kubernetes service. Loadbalancer pod also operns the firewall on PC VMs.

Question 12

explain different sections of envoy config file

Answer 12

Envoy config file has two major sections,
Listeners section - These list all the front end ip ports on which envoy listens for client connections. This also contains the name of the cluster where the connections should be sent to backend
Clusters section - This section contains the backend k8s ip and the nodeport ports to which the connections should be sent to in the hosts parameter. This section also contains health checks section on how to monitor if the backend is healthy before we can send the connections to.

Question 13

Explain different types of DNS and the domains they serve and the clients they serve

Answer 13

At the base k8s level we have coredns which services
svc.mspfqdn and pod.mspfqdn subdomains only. The clients for core dns will be mostly pods running in k8s cluster.
The next layer of DNS is the mspdns which serves mspfqdn domain, services running in k8s are exposed to external clients via entries in the mspfqdn domain. The clients for this DNS are external to the k8s cluster where the services are running, for example services running inside PC VM like atlas, aplos, prism etc…
The next layer of DNS is customer’s DNS. This serves all the domains except mspfqdn including any public fqdn’s. All customer applications that are not running PC VM will have to reach the services running inside CMSP k8s cluster via this external DNS.

Question 14

Why MSP DNS is very important for cluster function

Answer 14

MSP DNS acts as the DNS server that connects services running inside a Prism Central VM to the services that are running inside CMSP k8s cluster. Some examples of entries that help CMSP k8s services are zk.mspfqdn, idf.mspfqdn, ergon.mspfqdn these entries allow services inside CMSP k8s cluster to talk to PC services

Question 15

what is the use of lb-controller-map config map

Answer 15

This map contains the list of loadbalancers that the lb controller pod should configure when a new loadbalancer type service is created in k8s cluster. It has the authentication ssh keys to login to the envoy lb’s and configure the files.

Question 16

Where does the mspdns run and how is it run as

Answer 16

MSP dns runs as a pod in the CMSP k8s cluster, it is exposed via

Question 17

What is the use of monitor host and monitor ip in msp service registry output

Answer 17

monitor port and monitor host are used by service registry to send frequent health checks, if in case there is no response for the monitoring, then the service registry entries are deactivated so clients will not try to connect to a down load balancer. This is a way to monitor the lb’s front end ips. Normally lb’s have their own health monitoring for their backend ips but there is none for their front ips. That’s why a service registry is more power ful than a regular dns

Question 18

Explain how a dns resolution happens in CMSP for a MSP service url like iam-proxy.ntnx-base

Answer 18

Client (aplos,atlas) -> PC resolv.conf -> 127.0.0.1 (dnsmasq) -> One of the PC VM node port 31000 -> mspdns pod -> MSP service registry pod -> mspdns etcd db -> stateDB -> return to client

Question 19

What does enable_iamv2.py and disable_iamv2.py

Answer 19

It allows clients like Mercury to be informed if IAM v2 is enabled or disabled.

Question 20

List some of the reasons for pc to be running slow

Answer 20

objects connection leak
broadcom vxlan issue
slow backend disks on pe cluster
slow iam dns resolutions

Question 21

PC GUI up but login with user ‘admin’ fail with error: access denied

Answer 21

issue could be with iam pods not upgraded after pc upgrade. restart msp controller to resume the base services upgrade