Language model configuration

By default, V‑Blaze loads all licensed and installed language and acoustic models at startup. This default model loading is sufficient for most implementations, and in most cases, language model loading does not need to be further configured.

For cases where V‑Blaze has more installed models than it needs or than it can load at one time, customized model loading can be configured with the models.list file in /opt/voci.

Customized loading controls which models to load at startup, which models to load on demand as needed for transcription, and other related options like model reload time and the amount of system memory to keep available. This customization can be used to reduce startup time, optimize performance, and manage model memory allocation.

Customized loading is used for very fine-tuning of advanced implementations, and models.list should not be configured without consultation with or guidance from the Voci support team.

Demand-based loading is available for language models only; all available acoustic models load at startup.

models.list

The models.list file is located in /opt/voci and is organized into lines. There are two types of line: the global #config line and the model line. models.list should contain only one #config line, but it may contain multiple model lines.

The optional #config line is used for global configuration parameters that are applied when V‑Blaze loads any model.

Model lines are used for parameters applied when V‑Blaze loads the line's specified models. A model line may specify a single model, or multiple models using a wildcard. For example, specifying eng* captures all installed and available English models, such as eng-us, eng-eu, and eng-ca.

Important:

When determining the order of lines in models.list, begin with the most specific and end with the least specific model sets. The first matched configuration is used.

Parameters

Some parameters may be used for both #config and model lines. The following table lists all parameters and describes their behavior by context.


Name	Context	Description
`cycle`	`#config` and model	Defines the number of hours between periodic model reloads. Specifying any value for cycle indicates that models should be reloaded periodically to de-fragment memory and potentially improve performance. By default, the system does not periodically reload models.
`memfree`	`#config` only	Specifies an amount of host system memory in gigabytes to keep available. The ASR engine will attempt to unload the least used language models until the `memfree` specification is met. By default, there is no minimum, and the system will not unload any models and will not attempt to retain a free-memory buffer. Does not affect acoustic models.
`gpumempad`	`#config` only	Default 40. Specifies an amount of GPU memory in mebibytes to leave unallocated for each GPU when loading models. Use this option in the rare case that the autosize-calculated configuration doesn't load successfully.
`expire`	`#config` and model	Defines the amount of idle time in hours after which the system unloads a language model. By default, the system never unloads a model due to idle time. Does not affect acoustic models. Note: The information used to make expiration decisions can be retrieved from the `mlru` element via the /statusfull endpoint.
`all=load\|demand\|no` [INTERNAL ONLY]	`#config` only	[INTERNAL ONLY] Specifies a global loading rule for all available language models. If there is no `models.list` file, or if it exists and contains no model lines or `all` lines, the system will try to load every model that is installed. If `models.list` contains `#config all=demand`, then unspecified models default to `demand`, and specified models default to `load` If `models.list` contains `#config all=load`, then unspecified models default to `load`, and specified models default to `demand` If `models.list` contains `#config all=no`, then unspecified models are not loaded and specified models are loaded. This configuration is not recommended. The `all` parameter is an internal development shortcut. Instead of using `#config all`, use a model line and a wildcard, as in `eng`. For example, the following could be used to manage a large library of installed `eng` models: `engvoicemail no eng-xa demand eng load` In this example, specifying `eng* load` first would override the `eng-xa demand` and `engvoicemail no` lines, and all models would load.
`{model} load\|demand\|no`	model only	`{model} load` Load the model at system start-up. The default behavior for any model listed in `models.list` is `load`; as a result, this keyword is typically not necessary. `{model} demand` Load the model when needed for transcription jobs. Using `demand` for dynamic loading better supports a variety of models that may not be in constant use. Language models use a significant amount of host memory. Dynamic loading incurs a small, one-time, on-demand cost when lesser-used models are required instead of keeping those models loaded in memory at all times. Because language model loading is serial and can take a long time if many models are installed and enabled, dynamic loading may also speed up the service startup time for serving the most important language models. The demand option is commonly used in conjunction with an `expire` directive to configure models to load when needed, then unload them after they are not used for the specified period. Note: `demand` affects only the language model component. All acoustic models required for `load` and `demand` models are initialized at startup. `{model} no` Never load the specified language model(s). Matching acoustic models are not loaded unless needed for another load or demand model. Use to exclude particular models or domains from wildcard model lines.
`numgpus=N`	`#config` and model	Define the number of GPU processes for associated acoustic models. Acoustic processes are shared among all language models that use the same acoustic model. Explicit setting of `numgpus` can be used in some scenarios to increase acoustic model copies for more performance on highly performant GPU configurations, or to reduce copies for models that don't require full throughput. Typically, this parameter is used to tune models into available GPU space to optimize multiple- and cross-model performance. Note: If two language models use the same acoustic model, then the number of GPU processes used for that acoustic model is the higher of the two `numgpus` values. For example: `eng-us:financial demand numgpus=2 eng-us:callcenter demand numgpus=4` In the preceding example, the acoustic model loads with 4 GPU processes. When there is a conflict between `numgpus` values for the same acoustic model: The largest model-line value overrides any other model-line values Any model-line value overrides a #config-line value, even if the model-line value is smaller Note: By default, `numgpus` is calculated using the following formula: `roundup(min(maxdecoders,ncpus) / 4` `roundup()` Round up if the final value is fractional `min()` Select the lowest of the two values `maxdecoders` Number of language model decoding processes to start `ncpus` Number of logical CPUs. Identical to the value of `cpu_count` or the `CPU(s):` line in `lscpu` command output. If hyperthreading is not enabled, this value matches the number of physical cores.
`numgpus=cN`	`#config` and model	Load acoustic model(s) in CPU only. When GPU resources are available on the host, configuring CPU-only acoustic model loading results in the specified acoustic model(s) being enabled in host CPU and RAM rather than GPU. This configuration can be used to conserve GPU resources for models that require more performance while still making the specified models available for use at a lower throughput rate. On systems without GPU resources, the `numgpus=N` and `numgpus=cN` settings are equivalent.

Sizing and loading models into memory

Acoustic models are of fixed size and are always loaded at startup. Whether the model is loaded in GPU or CPU memory, the acoustic model process sizes do not change during ASR operation.

Language model processes, however, can be very dynamic in their use of system memory. They do not grow without bound, but there can be significant differences in language model memory usage between initialization and peak operation. Monitoring the memfree element in /statusfull or the asr_memory_low_bytes gauge in /metrics is recommended to avoid out-of-memory conditions due to over-sizing configurations .

Acoustic model auto-sizing when limited by GPU memory

If the system runs out of GPU memory when loading acoustic models, it restarts in safe mode, ends all acoustic model processes, and sets numgpus to 1. One of each installed and available model loads to measure the memory requirements and optimize loading behavior accordingly. The system restarts when it finishes calculating optimal memory sizing for the models, and attempts to pack as many models as it can into the GPU based on the initially requested configuration.

In this case the system logs the following entries to /var/log/vociserver/err.log during the memory sizing process:

Decode server failed to start. Restarting in safe mode.

Decode server successfully started in safe mode. Restarting.

Note: V‑Blaze attempts to adhere to the settings specified in models.list but may reduce some settings if it is unable to load all models with the specified configuration.

Example and default models.list

Figure 1. Standard models.list

#config memfree=5 cycle=24

The preceding code shows the generally recommended configuration for models.list that is typically included with Voci-produced system images like AWS EC2 AMIs.

Figure 2. Example: models.list with model lines only

eng-us:callcenter
fre-ca:callcenter

The preceding models.list configuration limits the models loaded at startup to eng-us:callcenter and fre-ca:callcenter.

Figure 3. Example: dynamic model loading

#config memfree=5 cycle=24 expire=12  
eng-us:callcenter load 
eng1* no
eng* demand

This example demonstrates dynamic loading.

The first line represents a global configuration command with the following settings:
- memfree=5 — ensures that there is at least 5 GB of free memory on the system.
- cycle=24 — reload models every 24 hours to help refresh memory.
- expire=12 — automatically unload any language model if it has not been used in 12 hours.
The second line instructs vociserver to automatically load eng-us:callcenter into memory when vociserver service starts up.
The third line ensures deprecated eng1 models are not loaded.
The fourth line uses a wildcard to load all other eng models dynamically.