Splunk Query Finds All Citrix XenDesktop Services

Splunk query to find status of Citrix XenDesktop Services

If you ever need to find all the Citrix services that run for XenDesktop in Splunk you can use this Splunk query. It will also give you the State of each service so you query if the service is running or not. I used this as the basis for many dashboards to report which Xen services are running on which servers and what is the current state.

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”Citrix*” OR Name=”ComTradeMPPVSAgent” OR Name=”CdfSvc” OR Name=”CtxLSPortSvc” OR Name=”CdfSvc” OR Name=”BNBOOTP” OR Name=”BNPXE” OR Name=”BNAbs” OR Name=”soapserver” OR Name=”StreamService” OR Name=”BNTFTP” OR Name=”PVSTSB” OR Name=”CtxAudioSvc” OR Name=”CtxFlashSvc” OR Name=”CtxSensVcSvc” OR Name=”MRVCSvc” OR Name=”cpsvc” OR Name=”ctxProfile” OR Name=”PvsVmAgent” OR Name=”ServicesManager” OR Name=”CtxSmartCardSvc” OR Name=”StackControlService” OR Name=”CitrixTelemetryService” StartMode=”Auto” State=”*” | sort Name, State | table host, Name, StartMode, State

How are you going to use this query?

If you modified please share the changes you made.

Splunk Query for Citrix PVS Services

Splunk Plus Citrix

If you have Citrix Xen Desktop in your environment you might be using PVS Server. Citrix PVS Server is a great way to decrease operational maintenance and VM density in your virtual environment. In case you want to use Splunk to analyze the data about your PVS environment here are the queries that I use. It is not defined by host so it will dynamically build the list for you based on the PVS services that are running on each host in your environment.

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CdfSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”ComTradeMPPVSAgent” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”BNBOOTP” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”BNPXE” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”BNAbs” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”soapserver” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”StreamService” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State
index=”windows”

eventtype=hostmon_windows Type=Service host=”*” Name=”BNTFTP” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”PVSTSB” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Event IDs 2000 to 2102 to Monitor with Splunk

Citrix Event ID 2000 to Citrix Event ID 2102

Citrix Event ID
Splunking Citrix? If you are or just need to figure out what the Event IDs are that Citrix is filling up your Windows Event Logs with here are Citrix Event IDs 2000 to 2102. These are more detailed than the earlier Event IDs and mainly have to do with the Citrix Broker Services. So if you are tracking down issues with XML services like the Citrix Broker Service this is range of Citrix Event IDs to look into.

Looking for Citrix Event IDs 1 to 509

Looking forĀ Citrix Event IDs 1000 to 1201

Event ID to Monitor
Event Message Text
Event ID 2000
The Citrix Profile management Group Policy Extension has started. Cause: The Citrix Profile management Group Policy Extension has started to process policies for user ‘Domain\user’. Action: This message is informational and no action is required.
Event ID 2001
The Citrix Broker Service failed to initialize XML services. The services will attempt to initialize again in approximately 1 minute(s).Exception ‘Input string was not in a correct format.’ of type ‘System.FormatException’.
Event ID 2003
The Citrix Broker Service successfully started XML services.
Event ID 2004
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: WerFault.exe (15428) consumed 6796857344 bytes, BrokerService.exe (9432) consumed 722894848 bytes, and BrokerService.exe (9528) consumed 637378560 bytes.
Event ID 2007
The Citrix Broker Service is stopping XML HTTP services.
Event ID 2008
The Citrix Broker Service successfully stopped XML HTTP services.
Event ID 2012
The Citrix Broker Service was unable to send a response to the XML client.
Event ID 2012
The Citrix Broker Service was unable to send a response to the XML client. Details: Request URL: ‘/SCRIPTS/WPNBR.DLL’ Error Code: ‘1229’ Error Message: ‘An operation was attempted on a nonexistent network connection’ Exception Type: ‘System.Net.HttpListenerException’ Exception Call Stack: ‘ at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size) at Citrix.Cds.Xms.Multiplexer.XmlMultiplexer.HandleRequest(HttpListenerContext context, DateTime startTime)’
An unexpected exception occurred while the Citrix Broker Service processed an HTTP request. Error details: Request URL: ‘https://ddc.Domain.com/scripts/wpnbr.dll’ Exception Type: ‘Citrix.Cds.Broker.DAL.DALPowerActionIgnoredException’ Exception Call Stack: ‘ at Citrix.Xms.XmlSupport.XmlPerf.TimeoutAction(Int32 timeoutMs, Action action) at Citrix.Cds.Xms.Wpnbr.WpnbrServer.HandleRequest(HttpListenerRequest request, WindowsIdentity identity, DateTime startTime)’
Event ID 2100
The Citrix Broker Service failed to validate a user’s credentials on an XML service. Verify the trust relationships between your domains. Error details: User: ‘Domain\timothy.frazier.wa’ Error: ‘InvalidCredentials’ Message: ‘Failed Windows logon, error code 50’ Stack Trace: ”
Event ID 2101
The Citrix Broker Service failed to validate a user’s credentials on an XML service again. A previously detected problem still exists. To avoid excessive event logging, the service is suppressing related messages (event ID 2100) until the problem is resolved. Initialization attempts will continue.
Event ID 2102
The Citrix Broker Service successfully validated user credentials. It is no longer suppressing the related messages (event ID 2100).

Splunk with btool for Combined Inputs.conf

Inputs not showing up in Splunk?

The other day I pointed some additional data at my Splunk Indexer but it was not showing up. I wanted to make sure I did not fat finger something or that I did not have overlapping inputs from other apps inputs. I found the following BTOOL command to run when you want to find out how Splunk has combined all the inputs.conf files that is it processing.

./splunk btool inputs list

Splunk BtoolYou run it from $SPLUNK_HOME/bin folder and it does not matter if you are on a Search Head, Indexer, DCN, or Universal Forwarder.
It outputs all the Stanzas in all the INPUTS.conf files in the final order they are processed in. You get to see which Stanzas won due to the folder and file precedence that Splunk uses when processing .conf files.

By the way if you want to get the same information about any .conf file other than inputs.conf you just replace place the name of .conf file after btool and before list.

If you want to save the results in a file so you can slice and dice the results in text editor or sort it with Excel use the following btool command.

./splunk btool inputs list > nameOfFile.txt

That list command will give you just the names of the Stanzas but what if you want to know the entire path to each file that contains those Stanzas?

Use this btool command that includes the list and –debug switches.

./splunk btool inputs list --debug > nameOfFile.txt

The output gives you the full path to each .conf file of each processed Stanza.

It is a lot of data about those little old Stanza that mean so much to Splunk and the data you sent your way.

Try btool for props.conf and transforms.conf it will help provide even more insight to your data and the is what it is all about.

Here is the link to the Splunk>docs btool page.

Here is the link to a great Splunk Blog page about btool

Splunk Query to Determine NetScaler HA Status

Determine Citrix NetScaler High Availability Cluster Status with this Splunk Query

I hope you have your NetScalers setup in a HA pair and if you do you might want to know which NetScaler is acting as the Primary Node, the NetScaler Cluster fail-over status, or even if the HA pair is UP. Maybe you want to monitor the High Availability status over time and digging through email alerts is not the way to go.

Here is a Splunk query to run if you have Splunk Add-on for Citrix NetScaler.

index="netscaler" source="stat:hanode" sourcetype="citrix:netscaler:nitro"
 failover_status="1" failover_status_string="UP" hacurmasterstate="Primary" hacurstate="UP" hacurstatus="YES" haerrsyncfailure="0"

Let’s break that code down.

So here is the part the defines the index. The default is “netscaler”; however, it might be different on your system so check your Indexes and inputs.conf to be sure.

index="netscaler"

The source for this NetScaler query is the “stat:hanode” that comes from the NITRO API data.

source="stat:hanode"

The sourcetype is “citrix:netscaler:nitro” which is the default sourcetype for the data coming in via the NITRO API calls to the NetScalers.

sourcetype="citrix:netscaler:nitro"

Next up we have the “failover_status” that gives you the Boolean result of the status of the HA pair.

failover_status="1"

Following is the “failover_status_string” which provides an UP/DOWN string instead of the Boolean 0/1.

failover_status_string="UP"

“hacurmasterstate” is the field to check out when you need to know which NetScaler is currently running the HA Pair: “Primary” or “Secondary”.

hacurmasterstate="Primary"

If you want to know what the current state of the HA pair is here is the field, “hacurstate”. It returns “UP/DOWN”.

hacurstate="UP"

I’m not 100% sure about this one only because I would not answer the question, “what is the HA Current Status?” with YES.

hacurstatus="YES"

This one is important to monitor and trend as it can predict the stability of the HA Pair. This field/value will tell you if the HA pair is generating sync failures.

haerrsyncfailure="0"

Splunkbase is where you can find the Splunk Add-on for Citrix NetScaler.