What is the Operation Mode Server?

The Operation Mode Server is responsible for monitoring the Premise Callback solution to ensure operation modes are changed. It "sees" when an operation mode should change based when one or more of these conditions occur:

• Manual operation mode changes from Configuration
• Scheduled operation modes
• Holiday operation mode

Operation Mode Server is a required application installed on each Core Instance server in a Callback solution.

Looking for more?

Architecture Overview

Details on each connection point:

1. Operation Mode Server, on a configurable interval, will check the VHT_Config database to see when operation modes are changed by a user in Configuration, changed due to a scheduled operation mode, and changed due to a holiday operation mode
2. Queue Manager, on a configurable interval, will ask the Operation Mode Server for the the current operation mode for each queue. Queue Manager will change the operation mode for queues in the Callback solution if Operation Mode Server returns a new active operation mode.
3. Operation Mode Server will inform Notification Server on whether it's started or stopped. This is information will be used to display the status of Operation Mode Server in the System Management UI.

Configuration UI Settings

These settings are used to determine what the current operation mode should be a queue. They are all found in the Configuration UI.

Operation Mode

Found in the Configuration UI under Call Flow -> Treatment, the Operation Mode setting is used to show the current operation mode set for the queue. This setting can be adjusted to manual change the operation mode.

Ignore Scheduled Operation Mode Events

Found in the Configuration UI under Call Flow -> Treatment, the Ignore Scheduled Operation Mode Events setting is used to prevent the changing of an operation mode based on the defined schedule for the queue. This setting is commonly off. This setting is usually invoked when the call center wants to prevent the registration or dialing of callbacks.

Startup Operation Mode

Found in the Configuration UI under Call Flow -> Treatment, the Startup Operation Mode settings is used by Operation Mode Server during startup to determine if the last set operation mode for the queue (Last Mode Set) or the current scheduled operation mode (Follow Operation Scheduled) should be used by a queue after restart. This is typically set to use the Last Mode Set.

Scheduled Operation Modes

Found in the Configuration UI under Schedules -> Operation Schedules, this setting is used to define the time of day operation modes should automatically change for each queue.

This example shows the queue changing to Normal mode at 8:00 AM, Queue Speak at 5:00 PM, and AfterHours at 6:00 PM Monday - Friday. There are no schedule operation mode changes for Saturday and Sunday. This will result in the queue staying in the AfterHours operation mode all weekend until 8:00 AM Monday morning when the queue will change to Normal Mode.

Smart Features are not controlled or triggered to be active by Operation Mode Server. This setting is controlled and managed by Queue Manager.

Holiday Operation

Configuration File for Operation Mode Server

Operation Mode Service reads a configuration file during startup. This configuration file can be found in the Callback install directory on each Core Instance server.

You can find the installation location by viewing the registry on the Core Instance Server. Operation Mode Server is a 32-bit application.This is the path to the registry setting.

• Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Virtual Hold\InstallationLocation

The install directory is where you will find the configuration file for the Operation Mode Server.

• VHT_OperationModeServer_Console.exe.xml

These tabs detail all configuration settings available in this file. Any changes to the settings does require a restart of Operation Mode Server.

Changing Operation Modes

There are 2 settings that have very similar names in the configuration.

• monitorOpModeDelay
• MonitorOpModeDelay

The capitalization of the letter 'm' is the only difference between these settings. The MonitorOpModeDelay setting is not used anymore by Operation Mode Server.

monitorOpModeDelayThe monitorOpModeDelay setting is used by Operation Mode Service to determine how often it should check the VHT_Config database to see if an operation mode is changed. This includes the checking the following conditions for every queue in the Callback solution.

• Manual operation mode changes Configuration
• Scheduled operation modes
• Holiday operation mode

This is an example from a functional configuration file

The value of the setting is set in seconds. This example shows the setting configured for 1 seconds. This would trigger Operation Mode Server to check the VHT_Config database to see if an operation mode needs to be changed for every queue, pause for 1 second, and then perform the check again. The cycle will continue to repeat while Operation Mode Service is active.

A restart of Operation Mode Server is required if you change the value of the monitorOpModeDelay setting.

Database Connection

The configuration file for Operation Mode Server contains information on how Operation Mode Server can connect to the database to monitor for operation mode changes.

The connection string in the configuration file can either define windows authentication or a SQL authentication to access the database.

• Windows Authentication
  • The service account assigned to the Core Monitor windows service will be used as the identity of the user when access the database.
  • SQL will have to have this user listed and defined with read and write access to Callback databases
• SQL Authentication
  • A user account managed within SQL.
  • The password for this account will not be encrypted in this file.

QMCL Settings for Queue Manager

QMCL is a message protocol defined within Callback for various applications to communicate DIRECTLY with Queue Manager. There is no "middle man" like MessageBus or RabbitMQ.QMCL settings is used by Operation Mode Server to establish a communication path for Queue Manager to ask on a defined interval what is the current operation mode.

• tenantName: The site name for the Callback Solution
• hloc: Unique identifier for Operation Mode Server to keep messages separate
• ipAddress: The IP address of Operation Mode Server
• port: The TCP port to be opened by Operation Mode Server for Queue Manager to use to ask what is the current operation modes

This setting is what tells Operation Mode Server which to TCP port to open for Queue Manager communication. Operation Mode Server must use a static port for this communication where Queue Manager can use a dynamic TCP port (0 - Let the OS pick an available port)

Log Levels

Default log level for Operation Mode server is log level 5. While this provides some high level views of common functions/roles of operation mode server, additional logging maybe necessary to help view more detailed information. Adjust log levels accordingly. A restart of the Operation Mode Server will be required.

1. The 'level' settings is where you change the log level. Values 0-10 are acceptable
2. The 'logFilePath' is the location of the logs files.

Change to these settings does require a restart of Operation Mode Server for the new settings to take effect.

Registry Settings for Operation Mode Server

There are 2 types of registry settings in place regarding Operation Mode Server.

Communicating with Queue Manager

The registry settings location on how Queue Manager should communicate with Operation Mode Server are located on the registry for each Core Instance Server.

• Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Virtual Hold\ QMCL Settings\VHTOperationMode

These settings are used by Queue Manager to determine how to contact Operation Mode Server to find out if operation modes have changed.

This is required to establish the QMCL communication between Queue Manager and Operation Mode Server. QMCL is a message protocol defined within Callback for various applications to communicate DIRECTLY with Queue Manager. There is no "middle man" like MessageBus or RabbitMQ.

Queue Manager asking for current opmode

The OperationModeIntervalregistry setting, set in seconds, is how often Queue Manager should ask Operation Mode Server for the current operation mode.

The location of the registry setting on each Core Instance server.

• Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Virtual Hold

Changing this setting will require a restart of Queue Manager.

Heartbeat with System Management

On a defined time interval, Operation Mode Server will send a message through RabbitMQ to Notification Server. This message is referred to as a "Heartbeat". The heartbeat tells Notification Server that Operation Mode Server is stating it is still running. It might not be operating correctly, but the application is started. This is how System Manager can display Operation Mode Server as running.

This is done through a registry setting.

• Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Virtual Hold

When this setting is not set, the default time is 5 seconds. The value of the setting is in milliseconds. In this example, the heartbeat with Notification Server is sent every 15 seconds.

This registry setting is used by other applications in the Callback solution.

Logging Examples for Common Actions

A quick tip on log reading

Here is an example of a logs from Operation Mode Server. The MSG## after the time stamps indicates the log level required to write the log statement. In this example log level is set to 7 or higher since log statements are from levels 3, 5, and 7.

2021-10-12 11:46:17.905|MSG03|Rabbitmq Heartbeat overidden by registry, interval set to 15000|12021-10-12 11:46:17.912|MSG05|HeartbeatGenerator attempting to get a new producer from the factory...|42021-10-12 11:46:18.600|MSG05|HeartbeatGenerator successfully obtained and initialized a new producer.|42021-10-12 11:46:18.604|MSG07|HeartbeatGenerator successfully sent a heartbeat message.|4

Startup of Operation Mode Server

The startup of Operation Mode Server does have log statements that indicate the initialization of Operation Mode Server is complete. There are a set of steps performed by Operation Mode Server during the startup that will be logged and should be checked to verify a successful startup.

These are the steps during the startup process.

Step 1 - Starting the Heartbeat Process

During startup of Operation Mode Server, log statements are found showing the initialization of the heartbeat process with Notification Server.

2021-10-12 11:46:17.905|MSG03|Rabbitmq Heartbeat overidden by registry, interval set to 15000|12021-10-12 11:46:17.912|MSG05|HeartbeatGenerator attempting to get a new producer from the factory...|42021-10-12 11:46:18.600|MSG05|HeartbeatGenerator successfully obtained and initialized a new producer.|42021-10-12 11:46:18.604|MSG07|HeartbeatGenerator successfully sent a heartbeat message.|4

The number of messages you will see depends on the log level. Here we can see log level 3, 5, and 7 statements.

Log level 3 shows the heartbeat interval starting at 15000 ms (15 seconds).

Log level 5 shows the heartbeat generation process starting

Log level 7 shows the heartbeat being successful sent to Notification Server.

This process is logged at the default log level of 5. You will see all of this except the sending of the heartbeat which is at log level 7.

Step 2 - Open the QMCL TCP port to talk with Queue Manager

An important step in startup is opening the defined Operation Mode Server TCP port to communicate with Queue Manager. This is logged at log level 10 and will not typically be seen since the default log level is 5.

2021-11-15 13:08:58.054|MSG10|Server IPAddres: 10.100.80.12 | Port: 9100|OperationModeServiceRunner|62021-11-15 13:08:58.061|MSG10|Server Host Address: 10.100.80.12|OperationModeServiceRunner|62021-11-15 13:08:58.151|MSG10|SocketManager|AddServerSocket|ConnectionID = 1|OperationModeServiceRunner|6

Step 3 - Determine the startup operation mode for each queue

The process determine the startup operation mode for each queue starts with building the database connection. This is shown at log level 10

2021-11-15 13:08:59.156|MSG10|Time for NHibernate to Configure: 00:00:00.6547255|OperationModeServiceRunner|62021-11-15 13:08:59.756|MSG10|Time for NHibernate to BuildSessionFactory: 00:00:00.6000285|OperationModeServiceRunner|62021-11-15 13:08:59.756|MSG10|Time to AddSessionFactoryToConnections: 00:00:00.0002481|OperationModeServiceRunner|6

Locks within the application are started to ensure process complete without interruption. This is done at log level 6.

2021-11-15 13:09:00.140|MSG06|Entering lock for CreateOperationModeQueues|OperationModeServiceRunner|62021-11-15 13:09:00.140|MSG06|Has lock for CreateOperationModeQueues|OperationModeServiceRunner|6

When checking each queue, Operation Mode Server looks for 3 things to determine what the current operation mode.

• Is Ignore Scheduled Operation Mode Events on or off?
• What was the last operation mode set?
• What is set for Start Up Operation Mode?

2021-11-15 13:09:00.209|MSG04|Locked Operation Mode for VHT-Sales is 'TRUE'|OperationModeServiceRunner|62021-11-15 13:09:00.210|MSG04|Initial Operation Mode for VHT-Sales set to '4'.|OperationModeServiceRunner|62021-11-15 13:09:00.210|MSG04|Locked Operation Mode for VHT-Sales is 'TRUE'|OperationModeServiceRunner|62021-11-15 13:09:00.210|MSG04|Operation Mode Initialization Method for VHT-Sales is ''|OperationModeServiceRunner|6

When Last Mode Set is the chosen Start Up Operation Mode settings, the log will show an empty string. It will have a is '1' when Follow Operation Scheduled is set.

2021-11-15 13:09:00.210|MSG04|Operation Mode Initialization Method for VHT-Sales is ''|OperationModeServiceRunner|62021-11-15 13:09:00.230|MSG04|Operation Mode Initialization Method for VHT-Service is '1'|OperationModeServiceRunner|6

You will not see what operation mode was selected based on these settings. You will have to look for the request for current operation modes from Queue Manager. This will be logged several seconds later at log level 10.

You will continue to see log statements like this for each queue in the Callback solution.

Step 4 - Startup Complete

The log statements showing the startup of Operation Mode Serve is complete are logged at log level 3.

2021-11-15 13:09:00.287|MSG03|Operation Mode Server started.|OperationModeServiceRunner|62021-11-15 13:09:00.297|MSG03|Operation Mode Service started|OperationModeServiceRunner|6

Sending a Heartbeat to Notification Server

Based on the HeartbeatInterval registry setting, the Operation Mode Server will log each time a heartbeat is sent to Notification Server.

2021-10-12 11:46:35.606|MSG07|HeartbeatGenerator successfully sent a heartbeat message.|4

This statement is only logged at log level 7. This log statement will write every 'x' seconds based on the configured heartbeat interval.

Determining current Opmode

Based on the monitorOpModeDelay setting in the configuration file for Operation Mode Server, a check for a change in operation modes occurs for every queue.

The first log statement is the kick-off of this process. These statements are logged at log level 3 & 4. With a default log level of 5, this is typically the only log statements you will see.

2021-11-10 09:13:21.140|MSG04|Checking Operation Modes.|VHT_OperationMode_Monitor|162021-11-10 09:13:21.140|MSG03|FindQueues for site VHT|VHT_OperationMode_Monitor|16

First up is finding a list of current queues. These statements are logged at log level 6.

2021-11-10 09:13:21.144|MSG06|Entering lock for FindAndAddNewQueues|VHT_OperationMode_Monitor|162021-11-10 09:13:21.144|MSG06|Has lock for FindAndAddNewQueues|VHT_OperationMode_Monitor|162021-11-10 09:13:21.144|MSG06|Released lock for FindAndAddNewQueues|VHT_OperationMode_Monitor|162021-11-10 09:13:21.144|MSG06|Entering lock for IsOperationModesQueuesValid|VHT_OperationMode_Monitor|162021-11-10 09:13:21.144|MSG06|Has lock for IsOperationModesQueuesValid|VHT_OperationMode_Monitor|162021-11-10 09:13:21.144|MSG06|Released lock for IsOperationModesQueuesValid|VHT_OperationMode_Monitor|162021-11-10 09:13:21.144|MSG06|Entering lock for DetermineCurrentOperationModesForAllQueues|VHT_OperationMode_Monitor|162021-11-10 09:13:21.144|MSG06|Has lock for DetermineCurrentOperationModesForAllQueues|VHT_OperationMode_Monitor|16

With a list queues known, a check for the current operation mode is done. Operation Modes Server checks for a manual, scheduled, or holiday operation mode change. Each queue will log these statements. This is logged at log level 6.

2021-11-10 09:13:21.148|MSG06|Start DetermineCurrentOperationMode for Queue Sales|VHT_OperationMode_Monitor|162021-11-10 09:13:21.150|MSG06|Getting NextHolidayOperationMode for Queue Sales|VHT_OperationMode_Monitor|162021-11-10 09:13:21.151|MSG06|Getting NextScheduleOperationMode for Queue Sales|VHT_OperationMode_Monitor|162021-11-10 09:13:21.152|MSG06|Finished DetermineCurrentOperationMode for Queue Sales|VHT_OperationMode_Monitor|16

A change in operation mode will be logged based on the type of operation mode change.

A manual operation mode change.

2021-11-10 10:33:19.965|MSG04|Operation Mode for VHT-Sales changed to '3', due to manual intervention.|VHT_OperationMode_Monitor|16

2021-11-10 10:00:00.128|MSG04|Operation Mode for VHT-Training_2 changed to '2', due to schedule.|VHT_OperationMode_Monitor|16

When Ignore Scheduled Operation Mode Events is on, Operation Mode Server will not change the operation mode based on the scheduled operation mode. There is no log statement to show the scheduled operation mode change not occurring due to this setting.

2021-11-10 10:37:02.579|MSG04|Operation Mode for VHT-Sales changed to '6', due to holiday.|VHT_OperationMode_Monitor|16

Queue Manager checking for current Opmode.

Queue Manager will use the OperationModeInterval registrysetting is used to determine how often to check for operation mode changes with Operation Mode Server.

The logging starts at receiving the QMCL message from Queue Manager. This is at log level 10 only.

2021-11-10 13:57:00.332|MSG10|Data Received with bytesRecieved = 57|2|152021-11-10 13:57:00.332|MSG10|SOCKET_RECEIVE: bytes [++57++]|2|152021-11-10 13:57:00.332|MSG10|2|DataSocket|ReceiveThread|Data|Length57|VHT_QMCL0700000000024<MsgId>213</MsgId><ReqId>175</ReqId>|2|15

Next up is processing the QMCL message from Queue Manager. This is at log level 6 and 10.

The MsgID is an internal numerical identifier to the question being asked. In this case, it's 213 or GETOPMODES is the ask.

The ReqID is a numerical value assigned to the request. You can use this to track the QMCL messages across logs

2021-11-10 13:57:00.332|MSG06|QMCL Message Received VHT_QMCL0700000000024<MsgId>213</MsgId><ReqId>175</ReqId>|3|82021-11-10 13:57:00.332|MSG10|2|TransportSocketManager|ProcessData|MSG: Signature=VHT_QMCL, Type=07, HLOC=00000, HexLength=000024, IntLength=36, HeaderStr=VHT_QMCL0700000000024|3|82021-11-10 13:57:00.332|MSG10|2|TransportSocketManager|ProcessData|Parse String: <MsgId>213</MsgId><ReqId>175</ReqId>|3|82021-11-10 13:57:00.332|MSG10|Received QMCL_REMOTE_PROC_REQUEST|3|82021-11-10 13:57:00.332|MSG06|Entering lock for IsOperationModesQueuesValid|3|82021-11-10 13:57:00.332|MSG06|Has lock for IsOperationModesQueuesValid|3|82021-11-10 13:57:00.332|MSG06|Released lock for IsOperationModesQueuesValid|3|82021-11-10 13:57:00.332|MSG06|Entering lock for FindOperationModesForAllQueues|3|82021-11-10 13:57:00.332|MSG06|Has lock for FindOperationModesForAllQueues|3|82021-11-10 13:57:00.332|MSG06|Released lock for FindOperationModesForAllQueues|3|8

Operation mode server will identify the operation mode for each queue. This is logged at log level 10. Each queue will log the current operation mode determined by operation mode server. It is logged as VHT|QueueID|OPMODE;

The highlighted example shows the Training_3 queue is in Queue Speak.

The ReqID from the processing of the QMCL message is found in this log statement too.

Numerical values for operation modes. This numerical values are reflected in logs and database configuration.

1. Normal
2. QSpeak
3. Purge
4. RapidPurge
5. Shutdown
6. Standby
7. AfterHours
8. VirtualQ
9. DateBook

There is a single log level 3 message showing the GetOpmodes request being processed here. This would be the only log statement found for this activity at the default log level.

2021-11-10 13:57:00.332|MSG10|2|DataSocket|Send|Length 279|VHT_QMCL0800002000102<Message><MsgId>VHRPC_GETOPMODES</MsgId><RetValue>VHT|BCBS_Training|1;VHT|Sales|4;VHT|Service|6;VHT|Training_1|6;VHT|Training_2|2;VHT|Training_3|2;VHT|Training_4|1;VHT|Training_5|1;VHT|Upgrade_Bronze|1;VHT|Sales_Gold|1;</RetValue></Message><ReqId>175</ReqId>|3|82021-11-10 13:57:00.332|MSG10|SOCKET_SEND bytes [--279--]|3|82021-11-10 13:57:00.332|MSG03|ProcessRPCRequest|Received GetOpModes request connnectionID = 0|3|8

This series of log statements will write again after the defined time period of the OperationModeInterval registry setting.