Utility functions

irtorch.utils.cross_validation(model: BaseIRTModel, data: Tensor, folds: int, params_grid: dict, theta_estimation: str = 'ML', device: str = 'cpu', cores_to_use: int = None, multiprocessing: bool = True, **kwargs) → DataFrame

Perform cross-validation on the given model and data. Uses log-likelihood for model evaluation. Note that for running on the CPU on windows, if __name__ == ‘__main__’: needs to be added to the main script before calling this function, see examples.

Parameters

• model (IRT) – The irt model to train. Note that this should be an untrained model.

• data (torch.Tensor) – The data to use for cross-validation. The data is randomly shuffled before splitting into folds.

• folds (int) – The number of folds to use for cross-validation.

• params_grid (dict) – The hyperparameters to use for cross-validation. All need to be arguments for the model fit method.

• theta_estimation (str, optional) – Method used to obtain the theta scores. Can be ‘NN’, ‘ML’, ‘EAP’ or ‘MAP’ for neural network, maximum likelihood, expected a posteriori or maximum a posteriori respectively. (default is ‘ML’)

• device (str, optional) – The device to use for training. Can be ‘cpu’ for CPU or ‘cuda’ for GPU (if available). The default is ‘cuda’ if a GPU is available, otherwise ‘cpu’.

• **kwargs – Additional keyword arguments to pass to the model fit method.

Returns

A list of dictionaries containing the hyperparameters and the corresponding cross-validation scores.

Return type

list

Examples

>>> # Import necessary modules, load the data and split it into a training and testing set:
>>> import irtorch
>>> from irtorch.models import GeneralizedPartialCredit
>>> from irtorch.estimation_algorithms import MML
>>> from irtorch.utils import split_data, cross_validation
>>> data_math = irtorch.load_dataset.swedish_national_mathematics_2()
>>> train_data, test_data = split_data(data_math, 0.8)
>>> # Initialize the IRT model:
>>> model = GeneralizedPartialCredit(data=data_math)
>>> # Set up a grid of parameters for cross-validation:
>>> params_grid = {
...     "learning_rate": [0.1, 0.15],
...     "learning_rate_update_patience": [3, 7],
... }
>>> # Perform cross-validation to find a good set of parameters:
>>> if __name__ == '__main__':
>>>     result = cross_validation(
...         model,
...         data=train_data,
...         folds=5,
...         params_grid=params_grid,
...         theta_estimation='ML',
...         device='cpu',
...         algorithm = MML(),
...         multiprocessing=True
...     )
>>>     print(result)
>>>     print(f"--------------------\nBest hyperparameters\n--------------------")
>>>     best_params = result.loc[result["log_likelihood"].idxmax(), ["learning_rate", "learning_rate_update_patience"]]
>>>     print(f"learning rate: {best_params['learning_rate']} \nlearning rate update patience: {best_params['learning_rate_update_patience']}")

irtorch.utils.fit_multiple_models_cpu(models: list[irtorch.models.base_irt_model.BaseIRTModel], train_data: Tensor, cores_to_use: int = None, **kwargs) → list[irtorch.models.base_irt_model.BaseIRTModel]

Train multiple models on the same data using multiprocessing.

Parameters

• models (BaseIRTModel) – The IRT models to train. Note that these should be untrained model instances.

• train_data (torch.Tensor) – The data to use for training.

• cores_to_use (int, optional) – Number of cores to use for multiprocessing when device is ‘cpu’. (default uses one per model)

• **kwargs – Additional keyword arguments to pass to the model fit method.

Returns

A list of trained models.

Return type

list[BaseIRTModel]

Examples

>>> from irtorch.models import GeneralizedPartialCredit
>>> from irtorch.estimation_algorithms import VAE
>>> from irtorch.load_dataset import swedish_national_mathematics_1
>>> from irtorch.utils import train_multiple_models_cpu
>>> data = swedish_national_mathematics_1()
>>> # train 3 models with 1, 2 and 3 latent variables
>>> models = [GeneralizedPartialCredit(latent_variables = i+1, data=data) for i in range(3)]
>>> if __name__ == '__main__':
...     train_multiple_models_cpu(models, data, algorithm = VAE())

irtorch.utils.gauss_hermite(n, mean, covariance)

Calculate the Gauss-Hermite quadrature points and weights for a multivariate normal distribution.

Parameters

• n (int) – Number of quadrature points.

• mean (torch.Tensor) – Mean of the distribution.

• covariance (torch.Tensor) – Covariance matrix of the distribution.

Returns

• x (torch.Tensor) – Quadrature points.

• w (torch.Tensor) – Quadrature weights.

Notes

To integrate a function against a multivariate Gaussian distribution, one can employ Gauss-Hermite quadrature with an appropriate change of variables. Beginning with the integral:

\[\int \frac{1}{\sqrt{\operatorname{det}(2 \pi \mathbf{\Sigma})}} e^{-\frac{1}{2}(\mathbf{y}-\boldsymbol{\mu})^{\top} \mathbf{\Sigma}^{-1}(\mathbf{y}-\boldsymbol{\mu})} f(\mathbf{y}) d \mathbf{y}\]

We use the transformation \(\mathbf{x}=\frac{1}{\sqrt{2}} \mathbf{L}^{-1}(\mathbf{y}-\boldsymbol{\mu})\), where \(\boldsymbol{\Sigma}=\mathbf{L} \mathbf{L}^{\top}\), leading to:

\[\int \frac{1}{\sqrt{\operatorname{det}(2 \pi \mathbf{\Sigma})}} e^{\mathbf{x}^{\top} \mathbf{x}} f(\sqrt{2} \mathbf{L} \mathbf{x}+\boldsymbol{\mu}) \operatorname{det}(\sqrt{2} \mathbf{L}) d \mathbf{x}=\int \pi^{-\frac{N}{2}} e^{-\mathbf{x}^{\top} \mathbf{x}} f(\sqrt{2} \mathbf{L} \mathbf{x}+\boldsymbol{\mu}) d \mathbf{x}\]

For an \(N\)-dimensional vector \(\boldsymbol{x}\), the integral can be decomposed into \(N\) nested Gauss-Hermite integrals, since the inner product in the exponent, \(\exp \left(\sum_{n=1}^N x_n^2\right)\), can be represented as a product.

Examples

Computing mean and variance of a multivariate normal distribution using 8 Gauss-Hermite quadrature points:

>>> import torch
>>> from irtorch.utils import gauss_hermite
>>> mu = torch.tensor([1, 0])
>>> cov = torch.tensor([[1.3, -0.213], [-0.213, 1.2]])
>>> points, weights = gauss_hermite(8, mu, cov)
>>> mean = torch.sum(weights[:, None] * points, dim=0)
>>> variance = torch.sum(weights[:, None] * (points - mu)**2, dim=0)
>>> print(f"Mean: {mean}")
>>> print(f"Variance: {variance}")

irtorch.utils.get_item_categories(data: Tensor)

Get the number of possible responses for each item in the data.

Parameters

data (torch.Tensor) – A 2D tensor where each row represents one respondent and each column represents an item. The values should be the scores/possible responses on the items, starting from 0. Missing item responses need to be coded as -1 or ‘nan’.

Returns

A list of integers where each integer is the number of possible responses for the corresponding item.

Return type

list

irtorch.utils.impute_missing(data: Tensor, method: str = 'zero', model: BaseIRTModel = None, mc_correct: list[int] = None, item_categories: list[int] = None) → Tensor

Impute missing values.

Parameters

• data (torch.Tensor) – A 2D tensor where each row is a response vector and each column is an item.

• method (str, optional) –

  The imputation method to use. Options are ‘zero’, ‘mean’, ‘random_incorrect’. (default is ‘zero’)

  • ’zero’: Impute missing values with 0.

  • ’mean’: Impute missing values with the item means.

  • ’random incorrect’: Impute missing values with a random incorrect response. This method is only valid for multiple choice data.

  • ’prior expected’: Impute missing values with the expected scores for the latent space prior distribution mean.

• model (BaseIRTModel, optional) – Only for method=’random_incorrect’ or ‘prior expected’. The IRT model to use for imputation. (default is None)

• mc_correct (list[int], optional) – Only for method=’random_incorrect’. A list of integers where each integer is the correct response for the corresponding item. If None, the data is assumed to be non multiple choice (or dichotomously scored multiple choice with only 0’s and 1’s). (default is None)

• item_categories (list[int], optional) – Only for method=’random_incorrect’. A list of integers where each integer is the number of possible responses for the corresponding item. If None, the number of possible responses is calculated from the data. (default is None)

irtorch.utils.imv(model1: BaseIRTModel, model2: BaseIRTModel, data: Tensor, model1_theta: Tensor = None, model2_theta: Tensor = None, max_iterations=100, **kwargs)

Quantifies the improvement in model predictions from model 2 over model 1 by computing the InterModel Vigorish (IMV) metric by Domingue et al. [6]. The IMV is the expected gain if one were to place fair bets on model 1 making correct predictions on the observed data, but the actual predictions are made by model 2.

Parameters

• model1 (BaseIRTModel) – The first model to compare.

• model2 (BaseIRTModel) – The second model to compare.

• data (torch.Tensor, optional) – The data to use for evaluating the models. Typically not the same as the model training data to avoid overfitting.

• model1_theta (torch.Tensor, optional) – The latent scores for the first model. If None, the latent scores are calculated using the first model. (default is None)

• model2_theta (torch.Tensor, optional) – The latent scores for the second model. If None, the latent scores are calculated using the second model. (default is None)

• max_iterations (int, optional) – The maximum number of iterations for the Newton-Raphson method. (default is 100)

• **kwargs – Additional keyword arguments used for theta estimation. Refer to irtorch.models.BaseIRTModel.latent_scores() for additional details.

Returns

The IMV value. A positive value indicates that model 2 predicts better than model 1.

Return type

torch.Tensor

Examples

>>> import irtorch
>>> from irtorch.estimation_algorithms import MML
>>> from irtorch.models import OneParameterLogistic, TwoParameterLogistic, ThreeParameterLogistic
>>> data = irtorch.load_dataset.swedish_sat_binary()[:, :80]
>>> train_data, test_data = irtorch.split_data(data, 0.8)
>>> model1 = OneParameterLogistic(train_data)
>>> model2 = TwoParameterLogistic(train_data)
>>> model3 = ThreeParameterLogistic(train_data)
>>> model1.fit(train_data, MML())
>>> model2.fit(train_data, MML())
>>> model3.fit(train_data, MML())
>>> imv12 = irtorch.utils.imv(model1, model2, test_data)
>>> imv23 = irtorch.utils.imv(model2, model3, test_data)
>>> imv13 = irtorch.utils.imv(model1, model3, test_data)
>>> print(f"IMV 1-2: {imv12:.4f}")
>>> print(f"IMV 2-3: {imv23:.4f}")
>>> print(f"IMV 1-3: {imv13:.4f}")

irtorch.utils.set_seed(seed: int)

Seed all random number generators for reproducibility (random, numpy and torch).

Parameters

seed (int) – The seed value to use for seeding the random number generators.

irtorch.utils.split_data(data, train_ratio=0.8, shuffle=True)

Splits a tensor into training and testing datasets.

Parameters

• data (torch.Tensor) – The dataset to be split. It should be a tensor where the first dimension corresponds to the number of samples.

• train_ratio (float, optional) – The proportion of the dataset to include in the train split. This should be a decimal representing the percentage of data used for training (e.g., 0.8 for 80% training and 20% testing). The default is 0.8.

• shuffle (bool, optional) – Whether to shuffle the data before splitting. It is highly recommended to shuffle the data to ensure that the training and testing datasets are representative of the overall dataset. The default is True.

Returns

A tuple containing two tensors. The first tensor is the training dataset, and the second tensor is the testing dataset. The split is based on the train_ratio parameter, and if shuffle is True, the split is performed on the shuffled data.

Return type

tuple[torch.Tensor, torch.Tensor]

Examples

>>> data = torch.rand(100, 10)  # Example tensor with 100 samples and 10 features each
>>> train_data, test_data = split_tensor(data, train_ratio=0.8, shuffle=True)
>>> print("Training data size:", train_data.shape)
>>> print("Testing data size:", test_data.shape)