DDC Service Status Splunk Query

Here are the Splunk queries that I use to to find the status of each Citrix XenDesktop service that runs on each DDC server. It is really useful to find all services but with these Splunk queries are break out each XenDesktop service into its own query so you can slice and dice depending on what you are looking for. It speeds things up in Splunk if you can narrow the amount of data you pull in the first place.

DDC Service Status Splunk Query

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixAdIdentityService ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixAnalytics ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixBrokerService ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixConfigurationLogging ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixConfigurationService ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixDelegatedAdmin ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixEnvTest ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixHostService ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixMachineCreationService ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixMonitor ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixPrivilegedService ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixStorefront ” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Splunk Queries for Citrix XenDesktop Services

I developed the following Splunk queries to count how many of each XenDesktop service is running in the Citrix Environment. Each query is based upon the Windows Service name and looks for the Start Mode to be AUTO. It will return a count of the service if the service is running or not. You only want to know the count if the service is running that is an easy change to the ‘State’ field value.

The Citrix EUEM service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”Citrix EUEM” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Encryption Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”Citrix Encryption Service” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Licensing service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”Citrix Licensing” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Peer Resolution Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”Citrix Peer Resolution Service” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix AD Identity Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixAdIdentityService” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Analytics service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixAnalytics” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Broker Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixBrokerService” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Configuration Logging service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixConfigurationLogging” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Configuration Replication service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixConfigurationReplication” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Configuration Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixConfigurationService” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Credential Wallet service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixCredentialWallet” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix CSE Engine service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixCseEngine” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Default Domain Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixDefaultDomainService” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Delegated Admin service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixDelegatedAdmin” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Environment Test service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixEnvTest” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Host Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixHostService” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Machine Creation Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixMachineCreationService” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Monitor service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixMonitor” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Privileged Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixPrivilegedService” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Redirector service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixRedirector” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Service Monitor service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixServiceMonitor” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Storefront service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixStorefront” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Subscriptions Store service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixSubscriptionsStore” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Telemetry Service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixTelemetryService” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

The Citrix Web Services for Licensing service:

index=”windows” source=”service” eventtype=hostmon_windows Type=”Service” host=”*” Name=”CitrixWebServicesforLicensing” StartMode=”Auto” State=”*”
| dedup host
| chart count(Name) as y

StoreFront Service Status by Splunk Query

Use Splunk  to Query StoreFront Service Status

Here are the Splunk Queries I use to find StoreFront Services in the XenDesktop environment. They are specific to each StoreFront service. The last SPL query will pull on the StoreFront services from hosts that you specify. Please let me know what you think and if they help in your XenDesktop environment.

Citrix Configuration Replication Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixConfigurationReplication” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Credential Wallet Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixCredentialWallet” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Default Domain Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixDefaultDomainService” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Peer Resolution Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”Citrix Peer Resolution Service” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Service Monitor:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixServiceMonitor” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Privileged Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixPrivilegedService” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Subscriptions Store Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixSubscriptionsStore” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Or if you want a single query to return the XenDesktop services status from a list hosts use this SPL Query:

index=”windows” eventtype=hostmon_windows Type=Service host=”StoreFrontServer01″ OR host=”StoreFrontServer02″ OR host=”StoreFrontServer03″ OR host=”StoreFrontServer04″ OR host=”StoreFrontServer05″ OR host=”StoreFrontServer06″ Name=”Citrix*” StartMode=”Auto” State=”*” | sort host, Name, State | table host, Name, State

XenDesktop Error Visualization in Splunk

Citrix XenDesktop Errors Visualized with Splunk

Here are the Splunk Queries I use when I need to create Splunk timechart dashboards to visualize Citrix XenDesktop errors. Try them out and let me know how they work in your environment.

This SPL Query grabs all Citrix related errors:

  • All Citrix Errors
    index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” error | timechart count(EventCode) by SourceName

This query displays database errors for the last 7 days:

  • All Citrix Database Errors (7 Days)
    index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” database* | timechart count(EventCode) by SourceName

Here is a query to display all events flagged as “failed” in the event log:

  • All Citrix Failures
    index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” failed| timechart count(EventCode) by SourceName

It is really not good to have timeouts anywhere in your Citrix XenDesktop environment so here is a timechart query to display when they are happening:

  • All Citrix Timeouts
    index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” timeout | timechart count(EventCode) by SourceName

Virtual machine events are easily displayed using this timechart query:

  • All Citrix Virtual Machine Events
    index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” virtual | timechart count(EventCode) by SourceName

Splunk StoreFront Dashboard

Need to create a StoreFront Dashboard in Splunk?Splunk-Citrix-Event-Visualzation

Here is a quick SPL query you can run to gather some basic information on that state of the Citrix XenDesktop services that are running in your environment. If you want to narrow it down to your StoreFront hosts just replace the asterisk in the “host=”*”” with the names of your StoreFront servers or the beginning pattern of the host names for you StoreFront servers.

 

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”Citrix*” StartMode=”Auto” State=”*” | sort Name, State | table host, Name, StartMode, State

Want to know how many of each Citrix service is running across those hosts?

Just add  | stats count by Name

Want to know how many Citrix services are running on each host?

Just add | stats count by host

 

 

Splunk REGEX for WinEventLog TerminalServices-LocalSessionManager

Splunk REGEX for RDP Session Logs

Ever want to get info on Terminal Services Local Session Manager Operations logs on your Windows servers to see who attempts to RDP into your Windows servers? Well here is a Splunk REGEX Field Extraction to get the RDP session info since Splunk was not able to figure out the field/value pairs on its own. For this REGEX to be useful you need to make sure that you are ingesting the WinEventLog for Microsoft-Windows-TerminalServices-LocalSessionManager/Operational. This Event Log captures all the events around RDP session creation, usage, and tear down. It is useful to tell who logged in using RDP and if any errors occurred during the session from start to finish.

In case you are wondering how I got to the below REGEX it is because Splunk’s REGEX engine has it’s own personality and the other REGEX tools online output “standard” REGEX that Splunk did not like.

WinEventLog:Microsoft-Windows-TerminalServices-LocalSessionManager/Operational : EXTRACT-TSOpsLog_Domain_UserName
^\d+/\d+/(\d+\s+)+\d+:\d+:(\d+\s+)+(\w+\s+)+\w+=\w+\-\w+\-\w+\-\w+/(\w+\s+)+\w+=\w+\-\w+\-\w+\-(\w+\s+)+\w+=(\d+\s+)+\w+=(\d+\s+)+\w+=(\w+\s+)+\w+=\w+\d+\w+\d+\.\w+\.(\w+\s+)+\w+=\w+_(\w+\s+)+\w+=\w+\-\d+\-\d+\-(\d+\s+)+\w+=(\d+\s+)+\w+=(\w+\s+)+\w+\.\s+\w+=(\w+\s+)+\w+=(\d+\s+)+\w+=(\w+\s+)+\w+=(\w+\s+)+\w+:\s+(\w+\s+)+\w+:\s+\w+:\s+(?P\w+\\)(?P\w+)

Splunk Query for Xen Desktop Services

If you have Citrix Xen Desktop in your environment and want to check on which server they are installed and their state use these Splunk queries. It is helpful so that you don’t have to keep an active list of the Xen Desktop servers. You could keep the list of Xen Desktop servers in a lookup but these queries are dynamic which saves you the headache of keeping the lookup current. These Splunk queries are also great if you are spinning up Xen servers on demand.

There is a Splunk query for each Citrix service that runs in a Xen Desktop environment so you can create a display for each one individually if you want.

XAV Server
Citrix Audio Redirection Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CtxAudioSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Desktop Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”BrokerAgent” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Diagnostic Facility COM Server
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CdfSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Encryption Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”Citrix Encryption Service” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix End User Experiencing Monitoring
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”Citrix EUEM” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Group Policy Engine
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixCseEngine” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix HDX MediaStream for Flash Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CtxFlashSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Location and Sensor Virtual Channel Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CtxSensVcSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Mobile Receiver Virtual Channel Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”MRVCSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Print Manager Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”cpsvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Profile Management
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”ctxProfile” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Pvs for VMs agent
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”PvsVmAgent” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Services Manager
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”ServicesManager” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Smart Card Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CtxSmartCardSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Stack Control Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”StackControlService” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

CitrixTelemetryService
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixTelemetryService” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Splunk Antivirus Exemptions

Splunk Antivirus Exemptions

splunk anti virus exemptionsThere are 2 main parts of your Splunk environment that will need exemptions from Antivirus software: the hosts that run Splunk and the hosts that run Universal Forwarder. The file locations are different for Windows and Linux. The file paths assume you have installed Splunk to the default location. If your environment required you to install to another directory/location just apply those changes to the paths below. You will also need to check and see if any of the scripts that are run as part of the apps or add-ons that you installed need to be exempted as well. This is the case in my environment and I have to get any file changes exempted before they will run in the production environment. This can get tricky when you are pushing changes to Universal Forwarders and the folders/files are transferred successfully but do not run to return results.

So as of the current release when this was written those are:

Splunk Enterprise on Windows Server

\Program Files\Splunk and all sub-directories
\Program Files\Splunk\var\lib\splunk and all sub-directories

Splunk Universal Forwarder on Windows OS

\Program Files\SplunkUniversalForwarder and all subdirectories

Splunk Enterprise on Linux

/opt/splunk and all sub-directories
/opt/splunk/var/lib/splunk and all sub-directories

Splunk Universal Forwarder on Linux OS

/opt/splunkforwarder and all subdirectories

Files to exclude on Windows

splunk-admon.exe
splunk-compresstool.exe
splunk-MonitorNoHandle.exe
splunk-netmon.exe
splunk-optimize-lex.exe
splunk-optimize.exe
splunk-perfmon.exe
splunk-regmon.exe
splunk-winevtlog.exe
splunk-winhostinfo.exe
splunk-winprintmon.exe
splunk-wmi.exe
splunk.exe
splunkd.exe
splunkweb.exe

Files to exclude on Linux

bloom
btool
btprobe
bzip2
cherryd
classify
exporttool
locktest
locktool
node
python*
splunk
splunkd
splunkmon
tsidxprobe
tsidxprobe_plo
walklex

 

Citrix Licensing Services Splunk Query

Find the Status of Citrix Licensing Services with Splunk

Citrix Licensing

During this exercise to get Splunk to ingest data from Citrix and make since of it we came to the Licensing server. We have ours running on a dedicated server; however,  the Splunk queries still search all hosts to keep the queries standard. Also if you ever planned on expanding your Citrix environment later you are covered. There are only 2 services that run on the license server concerned with Citrix: Citrix Licensing and CitrixWebServicesforLicensing.

These Splunk queries will give you a table with with hosts and the state of those 2 services. The dedup statement will prevent the same server from showing up more than once depending on the time frame your Splunk Universal Forwarder gathers data and the time frame your run your query for the dashboard.

index="windows" eventtype=hostmon_windows Type=Service host="*" Name="Citrix Licensing" 
StartMode="Auto" State="*" | dedup host | sort host, State | table host, State
index="windows" eventtype=hostmon_windows Type=Service host="*" Name="CitrixWebServicesforLicensing" 
StartMode="Auto" State="*" | dedup host | sort host, State | table host, State

As always please let me know what you think.

Thank you,
Logan Bingham

Citrix PVS Boot Time Splunk Query

Ever wonder how to get the boot time that Citrix XenDesktop PVS puts in the Windows Application Event Log in a format that you can calculate in Splunk?

I have my Windows Event Logs dumped into index=wineventlog with a sourcetype=WinEventLog:Application. When looking for the PVS bootime you need search for SourceName=StreamProcess. That will give you all the events for Stream Process and all of those events use EventCode=10 EventType=4 Type=Information. Unfortunately the way PVS records the boot time for each VM is in the Message field in the following format: Message=Device VMNAME boot time: X minutes Y seconds.

Here is my attempt that uses REGEX in the query to get the numbers out of the message field and into a table:

index="wineventlog" sourcetype=WinEventLog:Application  SourceName=StreamProcess Message="Device * boot time: * minutes * seconds." | rex field=_raw "(?ms)^\\d+/\\d+/\\d+\\s+\\d+:\\d+:\\d+\\s+\\w+\\s+\\w+=\\w+\\s+\\w+=\\w+\\s+\\w+=\\d+\\s+\\w+=\\d+\\s+\\w+=\\w+\\s+\\w+=\\w+\\d+\\w+\\d+\\.\\w+\\.\\w+\\s+\\w+=\\w+\\s+\\w+=\\w+\\s+\\w+=\\d+\\s+\\w+=\\w+\\s+\\w+=\\w+(?P<PVSDesktopName>\\s+\\w+\\s+)[^:\\n]*:\\s(?P<PVSBootTimeMin>[^\\s]+)\\s+\\w+\\s(?P<PVSBootTimeSec>[^\\s]+)" offset_field=_extracted_fields_bounds | table PVSDesktopName, PVSBootTimeMin, PVSBootTimeSec | sort -PVSBootTimeMin

Now you can work with the numbers to do some math.

index="wineventlog" sourcetype=WinEventLog:Application  SourceName=StreamProcess Message="Device * boot time: * minutes * seconds." | rex field=_raw "(?ms)^\\d+/\\d+/\\d+\\s+\\d+:\\d+:\\d+\\s+\\w+\\s+\\w+=\\w+\\s+\\w+=\\w+\\s+\\w+=\\d+\\s+\\w+=\\d+\\s+\\w+=\\w+\\s+\\w+=\\w+\\d+\\w+\\d+\\.\\w+\\.\\w+\\s+\\w+=\\w+\\s+\\w+=\\w+\\s+\\w+=\\d+\\s+\\w+=\\w+\\s+\\w+=\\w+(?P\\s+\\w+\\s+)[^:\\n]*:\\s(?P[^\\s]+)\\s+\\w+\\s(?P[^\\s]+)" offset_field=_extracted_fields_bounds | eval BootTimeSec=((PVSBootTimeMin*60)+(PVSBootTimeSec)) | table PVSDesktopName, BootTimeSec

Splunk PVS Boot Time In Seconds