chapter 20 Flashcards
PAM administration
Troubleshooting flow
Overview
the basic troubleshooting methodology for the PAM solution requires a thorough understanding of:
1. your system implementation
2. how components communicate with each other in your environment
3. What is the current behaviour compared to expected behaviour?
This methodology is designed to provide guidance and might not apply to every scenario
- it is important to write down any information gathered during this process and any tests performed as all of this information will be required when opening a case with CyberArk support
[none]
troubleshooting perquisites
- knowledge of the environment layout
- access to the different servers
- access to cyberark knowledgebase (customer community)
- access to cyberark documentation (publicly available online)
[none]
troubleshooting flow
- understand the environment’s topology
- initial questions, focus on user experience
- isolate the issue to specific scenario by trying to reproduce
- check relevant logs
- follow up questions
- check documentation and knowledgebase
- contact support
[none]
understanding the environment
Which components are installed and where?
what is the version of the relevant component?
is a load balancer being used?
are DR and HA solutions implemented?
[none]
Initial questions
initial questions, focus on user experience
user experience?
affected users?
error message displayed?
new implementation or worked and broken?
something changed when this issue started?
was there a process crash?
how does it impact production
Reproducible?
[none]
isolate the issue to specific scenario by trying to reproduce
Reproducible
- modify a variable and try to reproduce again
- repeat in different scenarios
- write down each scenario and the outcome of the test
- review the logs of reproduced scenarios (working and not working)
Not reproducible
- review the logs relevant for the reported flow
[none]
Checking the logs
log location
log types
log correlation
[none]
Follow up questions
Review and refine questions
[none]
Check documentation and knowledgebase
- Colleagues and end users
- knowledge base
- message and responses document; installation and implementation documents
- re-run scenarios
[none]
contacting cyberark support
- environment details?
- user experience?
- did it work in the past?
- are there any error messages?
- flow, current and expected behaviour?
- troubleshooting steps?
- steps to reproduce this issue?
- all relevant logs, screenshots and configuration files
[none]
troubleshooting flow
example
user unable to login
- unable to login to the privateark client using the administrator user
they see the message: ITATS004E authentication failure for user administrator
example topology:
1 vault prod
1 DR vault
1 PVWA, CPM, PSM
All running with version 12.6 on Windows 2019 servers
Initial questions:
is this issue experienced by all users?
- just one
did it work before?
- yes
was something changed?
- no
is there any error message?
- yes
isolate the issue to specific scenario by trying to reproduce
same issue with PVWA?
- yes
Reproducible?
- yes
check the relevant logs
error origin ITATS004E - ITA origin is vault, vault logs: ITAlog.log and Trace.d0
Check messages and responses
Try to identify the problem by searching in the “messages and responses” page in the online documentation
Home > Administration > References > Messages and responses
Messages displayed to end users are intentionally generic, listing many possible causes.
Because the error message starts with ITA, we know that the Vault server originated this error.
- at this point, we will go to the vault server and inspect the ITA log.
- there may be multiple log entries for the same problem
- try to find the first entry related to this problem
- when looking at the ITA log, we see error message ITATS528E with a code of 66
- when we search for that error, we see the exact cause of the problem and the solution
code 66 - the specified password is incorrect. Change it using the Change Password option
User unable to login
Solution
Does resetting the user password solve the problem?
Yes > Solved
No > check the relevant logs again - same error or a new one?
- repeat the troubleshooting flow
- contact support when no more logical steps are found
[none]
Types of logs:
console log - provides component-level entries such as service up or down
error log - exists in some components, and will include only error entries
trace log - provides detailed entries of workflows related to that component
debug logs - those logs may come in different types, sometimes they will be the trace files, with additional information and sometimes they will come at a dorm of separate files depending on the component
[none]
understanding cyberark logs
the log message code is built from 4 segments
EX:U8202026 ITA FW 001 I
ITA - the source component is the vault server
FW - the module with the message is the vault firewall
001 - message number
I - the message category
log messages are separated into four major categories
informational - ITAFW001I - firewall is open for client communication
warning - ITATS319W - firewall contains external rules
error - ITATS691E - LDAP synchronization error
system - ITADB367S - server unable to communicate with firewall
[none]
reviewing the logs
Once we get to a point where we need to go over log files, there
are a number of questions to ask:
* Which log file do we need to review?
* What do we search for?
⎼ Keywords (Error, Failed, Failure…)
⎼ Timestamps
⎼ User name
⎼ Object name (Account name, safe name)
* Are there correlated entries in other logs?
⎼ Log events and time of the issue
⎼ Different components
⎼ CyberArk logs and OS logs
[none]
set the debug mode for the vault ITAlog
the vault debug levels can be changed in the [blank].ini file, which will require a restart
the vault debug levels can be changed without a restart using the PARclient or Central administration station
dbparmi.ini
set the debug mode for the components
debug mode for components can be set in the configuration files stored on the vault or via the [blank] web UI
PVWA
log locations and configuring the debug levels
detailed information about setting debug level for different components and location of the log files can be found in the online documentation
setting vault log levels to [blank] should only be done under the guidance of cyberark support
debug
cheat sheet for the vault and related components
vault configuration file: dbparm.ini - changes require a vault restart
vault log file: italog.log
disaster recovery
config file: padr.ini
logs: padr.log
what can be used to troubleshoot the vault:
logic container - logiccontainer.log
PAReplicate.log - backup and restore
ene-event notification engine - eventnotificationengine.ini
PALog.txt
[none]
CPM Central Password Manager
Configuration File: Vault ➔ Safe “Password Manager”➔ root\policies<policy>.ini
Debug: PVWA ➔ Administration Tab ➔ CPM settings
CPMDebugLevels=2 (default)
0 – No messages will be written to the trace log.
1 – CPM exceptions will be written to the trace log (Default Level)
2 – CPM trace messages will be written to the trace log.
3 – CPM CASOS activities will be written to the trace log.
4 – CPM CASOS debug activities will be written to the trace log.
5 – CPM CASOS errors will be written to the trace log.
6 – All CPM CASOS activities and errors will be written to the trace log.
Logs – CPM: \Program Files\CyberArk\PasswordManager\Logs\pm.log
\Program Files\CyberArk\PasswordManager\Logs\pm-error.log\Program
Files\CyberArk\PasswordManager\Logs\PMConsole.log\Program
Files\CyberArk\PasswordManager\Logs\PMTrace.log
Logs –Plug-ins \Program Files\CyberArk|passwordManager\Logs\ThirdParty*.log
[none]
PSM Privileged Session Manager
Configuration File:
\Program Files\CyberArk\PSM\Basic_psm.ini
PVWA ➔ Administration Tab ➔ Options ➔ Privileged Session Management
Debug:
PVWA ➔ System tab ➔ Options ➔ Privileged Session Management ➔ General Settings
Server Settings ➔ TraceLevels=1,2,3,4,5,6,7
Recorder settings ➔ TraceLevels=1,2
Connection Client Settings ➔ TraceLevels=1,2
Logs: <installation>\Logs (and subfolders) or according to parameter
“LogsFolder” (located in Basic_psm.ini file</installation>
[none]
PVWA Password Vault Web Access
Configuration File :
\wwwroot\PasswordVault\web.config
Vault ➔ Safe “PVWAConfig” ➔
root\PVConfiguration.xml
Vault ➔ Safe “PVWAConfig” ➔ root\Policies.xml
Debug: PVWA ➔ Administration Tab ➔Options ➔ Logging
DebugLevel=High (options are None/High/Low/Profiling)
InformationLevel=High (options are None/High/Low/Profiling)
Logs:
%windir%\temp\
CyberArk.Webapplication.log
CyberArk.WebConsole.log
CyberArk.WebSession.<Sessionid>.log</Sessionid>
[none]
CyberArk xRay collects logs and configuration
files from PAM components in a simple, single step process
* The utility can be run from a remote machine or
on any of the CyberArk servers
* All data files are encrypted during collection,
regardless of whether they are collected locally or
remotely, and then transferred back to the xRay
machine
* You can share the collected data with your partner
or CyberArk, knowing that it is safely encrypted
during transfer
* When sharing with CyberArk, shared data is
linked to a case to allow Enterprise Support easy
and secure access to the collected data
can be downloaded from the cyberark marketplace
[none]
xray agent setup
- Select the component
- Select time frame for the collection and
collection level. - Select Collection scope
⎼ Logs from OS and the application
⎼ Logs from application only - Optionally, enable and provide the Active
Vault IP address and Administrative user
credentials for configuration files collection - Agree to the Terms of Use and click Start
Collection
You can monitor the collection
process as it collects the component
files
Once the process is complete, you
can select whether to:
⎼ Share the collected data with your
Partner
⎼ Share the collected data with
CyberArk
* You can also preview the data before
sending
* When sharing information with
CyberArk, make sure you have:
⎼ A Technical Community account
⎼ Case numbe
[none]
