How Slow is Slow?

Without a point of reference for comparison—a watched and ticking timer or a head-to-head race—most people won’t notice less than 33% difference in the time to complete a familiar task. A typical observer—one not actually timing things—faced with an hour-long task that completed in 40 minutes will think “hey, that seemed fast.” The same observer, waiting for a one second task to complete, will generally start frowning around 1,300ms.

StoreFront Service Status by Splunk Query

Use Splunk to Query StoreFront Service Status

Here are the Splunk Queries I use to find StoreFront Services in the XenDesktop environment. They are specific to each StoreFront service. The last SPL query will pull on the StoreFront services from hosts that you specify. Please let me know what you think and if they help in your XenDesktop environment.

Citrix Configuration Replication Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixConfigurationReplication” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Credential Wallet Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixCredentialWallet” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Default Domain Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixDefaultDomainService” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Peer Resolution Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”Citrix Peer Resolution Service” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Service Monitor:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixServiceMonitor” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Privileged Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixPrivilegedService” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Subscriptions Store Service:
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixSubscriptionsStore” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Or if you want a single query to return the XenDesktop services status from a list hosts use this SPL Query:

index=”windows” eventtype=hostmon_windows Type=Service host=”StoreFrontServer01″ OR host=”StoreFrontServer02″ OR host=”StoreFrontServer03″ OR host=”StoreFrontServer04″ OR host=”StoreFrontServer05″ OR host=”StoreFrontServer06″ Name=”Citrix*” StartMode=”Auto” State=”*” | sort host, Name, State | table host, Name, State

XenDesktop Error Visualization in Splunk

Citrix XenDesktop Errors Visualized with Splunk

Here are the Splunk Queries I use when I need to create Splunk timechart dashboards to visualize Citrix XenDesktop errors. Try them out and let me know how they work in your environment.

This SPL Query grabs all Citrix related errors:

All Citrix Errors
index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” error | timechart count(EventCode) by SourceName

This query displays database errors for the last 7 days:

All Citrix Database Errors (7 Days)
index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” database* | timechart count(EventCode) by SourceName

Here is a query to display all events flagged as “failed” in the event log:

All Citrix Failures
index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” failed| timechart count(EventCode) by SourceName

It is really not good to have timeouts anywhere in your Citrix XenDesktop environment so here is a timechart query to display when they are happening:

All Citrix Timeouts
index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” timeout | timechart count(EventCode) by SourceName

Virtual machine events are easily displayed using this timechart query:

All Citrix Virtual Machine Events
index=”wineventlog” sourcetype=”WinEventLog:Application” SourceName=”Citrix*” virtual | timechart count(EventCode) by SourceName

Create UEFI Bootable USB Flash Drive

UEFI Bootable Flash Drive Issues

I finally purchased a new laptop and of course it came with Windows Home pre-installed. I want to test out Windows Enterprise so I need to make a bootable USB flash drive with the Windows 10 Enterprise on it. I’ve used RUFUS for years to create bootable USB flash drives so without thinking I let it run with the defaults. Turns out with the new computers the BIOS is set to UEFI as default and the USB flash drive did not boot. After a little research I found information on which settings to use for RUFUS and there is a Microsoft tool specifically for creating UEFI bootable USB flash drives. The links to booth tools are below as well as to some really good how to guides for using both tools.

Microsoft Windows USB/DVD Download Tool

RUFUS

Creating Windows UEFI Boot-Stick in Windows by Thomas Krenn

How to install Windows 10 from USB with UEFI support by Windows Central

Splunk StoreFront Dashboard

Need to create a StoreFront Dashboard in Splunk?

Here is a quick SPL query you can run to gather some basic information on that state of the Citrix XenDesktop services that are running in your environment. If you want to narrow it down to your StoreFront hosts just replace the asterisk in the “host=”*”” with the names of your StoreFront servers or the beginning pattern of the host names for you StoreFront servers.

index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”Citrix*” StartMode=”Auto” State=”*” | sort Name, State | table host, Name, StartMode, State

Want to know how many of each Citrix service is running across those hosts?

Just add | stats count by Name

Want to know how many Citrix services are running on each host?

Just add | stats count by host

Splunk REGEX for WinEventLog TerminalServices-LocalSessionManager

Splunk REGEX for RDP Session Logs

Ever want to get info on Terminal Services Local Session Manager Operations logs on your Windows servers to see who attempts to RDP into your Windows servers? Well here is a Splunk REGEX Field Extraction to get the RDP session info since Splunk was not able to figure out the field/value pairs on its own. For this REGEX to be useful you need to make sure that you are ingesting the WinEventLog for Microsoft-Windows-TerminalServices-LocalSessionManager/Operational. This Event Log captures all the events around RDP session creation, usage, and tear down. It is useful to tell who logged in using RDP and if any errors occurred during the session from start to finish.

In case you are wondering how I got to the below REGEX it is because Splunk’s REGEX engine has it’s own personality and the other REGEX tools online output “standard” REGEX that Splunk did not like.

WinEventLog:Microsoft-Windows-TerminalServices-LocalSessionManager/Operational : EXTRACT-TSOpsLog_Domain_UserName ^\d+/\d+/(\d+\s+)+\d+:\d+:(\d+\s+)+(\w+\s+)+\w+=\w+\-\w+\-\w+\-\w+/(\w+\s+)+\w+=\w+\-\w+\-\w+\-(\w+\s+)+\w+=(\d+\s+)+\w+=(\d+\s+)+\w+=(\w+\s+)+\w+=\w+\d+\w+\d+\.\w+\.(\w+\s+)+\w+=\w+_(\w+\s+)+\w+=\w+\-\d+\-\d+\-(\d+\s+)+\w+=(\d+\s+)+\w+=(\w+\s+)+\w+\.\s+\w+=(\w+\s+)+\w+=(\d+\s+)+\w+=(\w+\s+)+\w+=(\w+\s+)+\w+:\s+(\w+\s+)+\w+:\s+\w+:\s+(?P\w+\\)(?P\w+)

Splunk Query for Xen Desktop Services

If you have Citrix Xen Desktop in your environment and want to check on which server they are installed and their state use these Splunk queries. It is helpful so that you don’t have to keep an active list of the Xen Desktop servers. You could keep the list of Xen Desktop servers in a lookup but these queries are dynamic which saves you the headache of keeping the lookup current. These Splunk queries are also great if you are spinning up Xen servers on demand.

There is a Splunk query for each Citrix service that runs in a Xen Desktop environment so you can create a display for each one individually if you want.

XAV Server
Citrix Audio Redirection Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CtxAudioSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Desktop Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”BrokerAgent” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Diagnostic Facility COM Server
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CdfSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Encryption Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”Citrix Encryption Service” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix End User Experiencing Monitoring
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”Citrix EUEM” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Group Policy Engine
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixCseEngine” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix HDX MediaStream for Flash Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CtxFlashSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Location and Sensor Virtual Channel Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CtxSensVcSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Mobile Receiver Virtual Channel Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”MRVCSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Print Manager Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”cpsvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Profile Management
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”ctxProfile” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Pvs for VMs agent
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”PvsVmAgent” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Services Manager
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”ServicesManager” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Smart Card Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CtxSmartCardSvc” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

Citrix Stack Control Service
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”StackControlService” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

CitrixTelemetryService
index=”windows” eventtype=hostmon_windows Type=Service host=”*” Name=”CitrixTelemetryService” StartMode=”Auto” State=”*” | dedup host | sort host, State | table host, State

PowerShell Commands to Install Hyper-V

Install Hyper-V using Windows PowerShell

install-windowsfeature –name hyper-v -includemanagementtools –restart

Install Hyper-V using DISM.exe, run the following command from an elevated command prompt:
dism /online /enable-feature /featurename:microsoft-hyper-v

Install the management tools with Windows PowerShell, you use the Install-Windows-Feature cmdlet, as follows:
install-windowsfeature -name rsat-hyper-v-tools

Install just the Hyper-V Manager or just the Hyper-V PowerShell module, you use one of the following commands:
install-windowsfeature -name hyper-v-tools
install-windowsfeature -name hyper-v-powershell

NetScaler Traffic Management Features

Here are the traffic management features of the Citrix NetScaler:

Cache Redirection

NetScaler Traffic Managaement analyzes incoming requests and forwards the requests for already cached data to cache servers. Dynamic HTTP requests and non-cacheable requests are forwarded to the origin servers.

Content Switching

Analyzes client requests and redirects the requests to specific servers on the basis of geographical area, authorization credentials, and device from which the request was initiated.

DataStream

Ensures optimal distribution of traffic from the application and web servers to the database servers. Enables you to segment traffic according to information in the SQL query and on the basis of database names, user names, character sets, and packet size.

Domain Name System

Provides authoritative domain name server (ADNS server) functionality for a domain. The NetScaler appliance functions as a DNS end resolver and forwarder, and also helps in name resolution when fully qualified domain names are not configured.

Firewall Load Balancing

Distributes the traffic across multiple firewalls, providing fault tolerance, increased throughput, and high availability.

Global Server Load Balancing

Enables disaster recovery and ensures continuous availability of applications by protecting against points of failure in a wide area network (WAN).

Link Load Balancing

Load balances outbound traffic across multiple Internet connections to transmit packets seamlessly over the best possible link.

Load Balancing

Distributes user requests for web pages and other protected applications across multiple servers to prevent server overloading and failure. Load balancing also provides fault tolerance.

SSL Offload and Acceleration

Offloads SSL processing from a server to the NetScaler appliance to accelerate SSL transactions.

Citrix PVS Boot Time Splunk Query

Ever wonder how to get the boot time that Citrix XenDesktop PVS puts in the Windows Application Event Log in a format that you can calculate in Splunk?

I have my Windows Event Logs dumped into index=wineventlog with a sourcetype=WinEventLog:Application. When looking for the PVS bootime you need search for SourceName=StreamProcess. That will give you all the events for Stream Process and all of those events use EventCode=10 EventType=4 Type=Information. Unfortunately the way PVS records the boot time for each VM is in the Message field in the following format: Message=Device VMNAME boot time: X minutes Y seconds.

Here is my attempt that uses REGEX in the query to get the numbers out of the message field and into a table:

index="wineventlog" sourcetype=WinEventLog:Application  SourceName=StreamProcess Message="Device * boot time: * minutes * seconds." | rex field=_raw "(?ms)^\\d+/\\d+/\\d+\\s+\\d+:\\d+:\\d+\\s+\\w+\\s+\\w+=\\w+\\s+\\w+=\\w+\\s+\\w+=\\d+\\s+\\w+=\\d+\\s+\\w+=\\w+\\s+\\w+=\\w+\\d+\\w+\\d+\\.\\w+\\.\\w+\\s+\\w+=\\w+\\s+\\w+=\\w+\\s+\\w+=\\d+\\s+\\w+=\\w+\\s+\\w+=\\w+(?P<PVSDesktopName>\\s+\\w+\\s+)[^:\\n]*:\\s(?P<PVSBootTimeMin>[^\\s]+)\\s+\\w+\\s(?P<PVSBootTimeSec>[^\\s]+)" offset_field=_extracted_fields_bounds | table PVSDesktopName, PVSBootTimeMin, PVSBootTimeSec | sort -PVSBootTimeMin

Now you can work with the numbers to do some math.

index="wineventlog" sourcetype=WinEventLog:Application  SourceName=StreamProcess Message="Device * boot time: * minutes * seconds." | rex field=_raw "(?ms)^\\d+/\\d+/\\d+\\s+\\d+:\\d+:\\d+\\s+\\w+\\s+\\w+=\\w+\\s+\\w+=\\w+\\s+\\w+=\\d+\\s+\\w+=\\d+\\s+\\w+=\\w+\\s+\\w+=\\w+\\d+\\w+\\d+\\.\\w+\\.\\w+\\s+\\w+=\\w+\\s+\\w+=\\w+\\s+\\w+=\\d+\\s+\\w+=\\w+\\s+\\w+=\\w+(?P\\s+\\w+\\s+)[^:\\n]*:\\s(?P[^\\s]+)\\s+\\w+\\s(?P[^\\s]+)" offset_field=_extracted_fields_bounds | eval BootTimeSec=((PVSBootTimeMin*60)+(PVSBootTimeSec)) | table PVSDesktopName, BootTimeSec