gemicai package Gemicai 050 documentation

gemicai.Classifier module

This module contains a Classifier class which simplifies model training and evaluation process by abstracting away
many implementation details. As a result this module allows the user to save a lot of time by providing a default
implementation for many of the PyTorch options.

Bases: object

This class does all of the heavy lifting when it comes down to the model training, evaluation and tensor
classification. During creation of this class it is possible to specify the following attributes:

module (nn.Module) – specifies a model to train, for more information about models themselves please refer
to the https://pytorch.org/docs/stable/torchvision/models

classes (list) – a list of classes present in the dataset, this will be used in order to modify model’s last layer.
For more information about how to obtain such a list please refer to the classes method of the
gemicai.data_iterators.DicomoDataset

layer_config (Optional[]) – optional parameter containing a functor that can be used to modify a given model.
For more information please refer to the gemicai.classifier_functors module

loss_function (Optional[nn.Module]) – optional parameter containing a loss function used during a training.

optimizer (Optional[torch.optim.Optimizer]) – optional parameter containing an optimizer used during a training.

enable_cuda (Optional[bool]) – if set to True the training will be done on the gpu otherwise the model will be
trained on the cpu. Please note that training on a gpu is substantially faster.

cuda_device (Optional[int]) – allows for selection of a particular cuda device if enable_cuda is set to True. PyTorch’s
Device ids start at 0 and are incremented by one.

RuntimeError – raised if the cuda device was selected but its not supported by the underlying machine

TypeError – raised if any of the parameters is of an invalid type

Takes in a tensor object and returns a list of predicted class types along with their certainties.

tensor (torch.Tensor) – tensor to classify

list of predicted classes and their certainty

TypeError – raised if tensor does not have a torch.Tensor type

Used to evaluate the model’s performance on a provided dataset.

dataset () – dataset iterator used in order to evaluate a model’s performance.

batch_size (int) – number (non-negative) of DataObject which will be feed into a classifier at once

num_workers (int) – number (non-negative) of worker threads used to load data from the dataset

pin_memory (bool) – whenever memory pages should be pinned or not. If set to false there is a possibility
that memory pages might be moved to a swap decreasing overall program’s performance.

verbosity (int) – specifies verbosity (non-negative) of training/evaluation output. 0 - no output, 1 - basic
output, 2 or more - extended output

output_policy () – specifies how and where to write the evaluation statistics

tuple of model’s accuracy, number of total images and number of correctly classified images

TypeError – if passed arguments are not of a correct type or their values are outside of valid
bounds this method will raise a TypeError exception.

ValueError – thrown whenever given data iterator object returns object containing more than two entries

Used to load a Classifier object from a given file

file_path (str) – a valid path to a file up to and including it’s extension type.

zipped (bool) – whenever given file is zipped or not

a valid Classifier object

TypeError – thrown if the given path is of an invalid format

Exception – thrown if Classifier object could not have been loaded in from the given file

Saves current classifier object to the file system, it can be loaded back in using the
gemicai.Classifier.Classifier.from_file method.

file_path (str) – a valid path to a file, it does not require a file to exist. Optionally .gemclas file
extension can be appended to a file path like so /home/test/classifier.gemclas, if the extension is not
present it will be added automatically.

zipped (bool) – whenever this object should be zipped or not

TypeError – file_path is not a str type

Used in order to select a device on which model training will be done.

enable_cuda (Optional[bool]) – if set to True the training will be done on the gpu otherwise the model will be
trained on the cpu. Please note that training on a gpu is substantially faster.

cuda_device (Optional[int]) – allows for selection of a particular cuda device if enable_cuda is set to True. PyTorch’s
Device ids start at 0 and are incremented by one.

RuntimeError – raised if the cuda device was selected but its not supported by the underlying machine

TypeError – raised if any of the parameters has an invalid type

Sets specified layers to be either trainable or not.

layers (list) – list of tuples specifying which layers should be trainable or not, eg. [(‘fc’, True), …]. Where
‘fc’ is a layer name and True specifies that it should be trainable. Note that instead of a layer
name it is possible to pass ‘all’ in its place which will set every layer in the model to the
specified mode, eg. [(‘all’, False)] makes every layer untrainable.

TypeError – thrown if layers parameter has a wrong type

Used to train a model.

dataset () – dataset iterator used in order to train a model

batch_size (int) – number (non-negative) of DataObject which will be feed into a classifier at once

epochs (int) – specifies how many training iterations (non-negative) to perform. One iteration goes over a
whole dataset.

num_workers (int) – number (non-negative) of worker threads used to load data from the dataset

pin_memory (bool) – whenever memory pages should be pinned or not. If set to false there is a possibility
that memory pages might be moved to a swap decreasing overall program’s performance.

verbosity (int) – specifies verbosity (non-negative) of training/evaluation output. 0 - no output, 1 - basic
output, 2 or more - extended output

test_dataset (Union[None, ]) – optional parameter, if a test_dataset iterator is passed and verbosity is set to at least 2
it will be used in order to evaluate model’s performance after a training epoch.

output_policy () – specifies how and where to write the training statistics

TypeError – if passed arguments are not of a correct type or their values are outside of valid
bounds this method will raise a TypeError exception.

ValueError – thrown whenever given data iterator object returns object containing more than two entries

Called internally in order to validate passed arguments to the train and evaluate methods.

dataset () – dataset iterator used in order to train/evaluate a model

batch_size (int) – number (non-negative) of DataObject which will be feed into a classifier at once

num_workers (int) – number (non-negative) of worker threads used to load data from the dataset

pin_memory (bool) – whenever memory pages should be pinned or not. If set to false there is a possibility
that memory pages might be moved to a swap decreasing overall program’s performance.

test_dataset (Union[None, ]) – optional parameter, validates whenever a test_dataset iterator passed to the train
function is a valid gemicai object

verbosity (int) – specifies verbosity (non-negative) of training/evaluation output. 0 - no output, 1 - basic
output, 2 or more - extended output

output_policy () – specifies how and where to write the training/evaluation output

epochs (int) – specifies how many training iterations (non-negative) to perform. One iteration goes over a
whole dataset.

TypeError – if passed arguments are not of a correct type or their values are outside of valid
bounds this method will raise a TypeError exception.

ValueError – thrown whenever given data iterator object returns object containing more than two entries

gemicai.ClassifierTree module

Bases: object

Bases: object

gemicai.classifier_functors module

This module contains functors that can be used to modify models. Object of this class can be passed as an optional
argument to the gemicai.Classifier.Classifier constructor, see it’s layer_config parameter for more information.

Bases:

Gemicai’s default functor which modifies model’s final layer depending on the number of passed classes.
It works with most of the torchvision models, for more information about models themselves please refer
to the https://pytorch.org/docs/stable/torchvision/models

Bases: abc.ABC

Every custom functor should extend this abstract base class.

gemicai.data_iterators module

This module contains data iterators which are used in order to traverse a dataset and retrieve a relevant information
from the DataObjects it contains.

Bases:

This class server as a proxy for the underlying iterators when the iter() method is called it returns a
PickledDicomoFilePool object which can be iterated over, this results in a class that supports a parallel data
loading. It’s constructor takes in the following parameters:

base_path (str) – a valid path to a folder containing a .gemset datasets

labels (list) – labels specifying which DataObject values except for a tensor will be returned by the next() call

transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor

constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method

TypeError – raised if any of the parameters has an invalid type

This iterator supports a parallelized resource loading.

always returns True

Plots one image per value type.

label (str) – label according to which we will look for a unique values, eg, ‘Modality’

cmap (str) – color scheme

TypeError – raised whenever label is not a str

Returns a dataset subset using provided constraints

constraints (dict) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

a valid gemicai.data_iterators.ConcurrentPickledDicomObjectTaskSplitter object

TypeError – raised whenever constraints parameter is not a dict

Returns or prints a summary of all the DataObject values in the dataset selected by the label.

label (str) – field label which values to summarize, for example ‘BodyPartExamined’ or ‘Modality’

print_summary (bool) – whenever to print or return an instance of gemicai.label_counters.GemicaiLabelCounter
object

if print_summary is set to false a class that extends a gemicai.label_counters.GemicaiLabelCounter

TypeError – raised whenever one of the parameter has an invalid type

Bases:

Every provided non-abstract data iterator extends this class and calls it’s __init__ method with a following
argument:

label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method

TypeError – raised if any of the parameters has an invalid type

Should return a boolean specifying whenever current iterator supports parallelized resource loading.

Returns a list of all of the classes in the dataset.

label (str) – label to summarize on

list of possible label values present in the dataset

Creates a data iterator from the supplied folder which should contain .gemset data sets

folder_path (str) – a valid path to an existing folder which contains .gemset datasets

labels (Optional[list]) – labels specifying which DataObject values except for a tensor will be returned by the next() call

transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor

constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

a valid gemicai.data_iterators.ConcurrentPickledDicomObjectTaskSplitter object

NotADirectoryError – raised whenever passed folder_path is invalid

Creates a data iterator for a supplied .gemset file

file_path (str) – a valid path to a .gemset file

labels (Optional[list]) – labels specifying which DataObject values except for a tensor will be returned by the next() call

transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor

constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

a valid gemicai.data_iterators.PickledDicomoDataSet object

FileNotFoundError – raised whenever file_path does not point to any valid file

Created a data iterator from the supplied file or folder path

data_set_path – a valid path to an existing folder which contains .gemset datasets
or a valid path to a .gemset file

labels (Optional[list]) – labels specifying which DataObject values except for a tensor will be returned by the next() call

constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

gemicai.data_iterators.PickledDicomoDataSet object if file path was supplied otherwise
gemicai.data_iterators.ConcurrentPickledDicomObjectTaskSplitter object

FileNotFoundError – raised whenever file_path does not point to any valid file

NotADirectoryError – raised whenever passed folder_path is invalid

Plots one image per value type.

label (str) – label according to which we will look for a unique values, eg, ‘Modality’

cmap (str) – color scheme

TypeError – raised whenever label is not a str

Should return a subset of a current dataset.

constraints (dict) – dictionary with a dataset constraints, eg. ‘Modality’: ‘CT’

a correct user defined iterator type which extends gemicai.data_iterators.GemicaiDataset

Returns or prints a summary of all the DataObject values in the dataset selected by the label.

label (str) – field label which values to summarize, for example ‘BodyPartExamined’ or ‘Modality’

print_summary (bool) – whenever to print or return an instance of gemicai.label_counters.GemicaiLabelCounter
object

if print_summary is set to false a class that extends a gemicai.label_counters.GemicaiLabelCounter

TypeError – raised whenever one of the parameter has an invalid type

Bases: abc.ABC, torch.utils.data.dataset.IterableDataset

This interface class serves as a basis for the every Gemicai’s data iterator.

Should return a boolean specifying whenever current iterator supports parallelized resource loading.

Should return a list of all the classes in the dataset.

label (str) – label to summarize on

list of possible label values present in the dataset

Should plot one image per class.

label (str) – label according to which we will look for a unique values

cmap (str) – color scheme

Should return a subset of a current dataset.

constraints (dict) – dictionary with a dataset constraints, eg. ‘Modality’: ‘CT’

a correct user defined iterator type which extends gemicai.data_iterators.GemicaiDataset

Should return or print a summary of all the DataObject values in the dataset selected by the label.

label (str) – field label which values to summarize, for example ‘BodyPartExamined’ or ‘Modality’

print_summary (bool) – whenever to print or return an instance of gemicai.label_counters.GemicaiLabelCounter
object

if print_summary is set to false a class that extends a gemicai.label_counters.GemicaiLabelCounter

Bases:

This class takes in a path to a folder containing a .gemset datasets and iterates over them.
It’s constructor takes in the following parameters:

base_path (str) – a path to a valid folder containing a .gemset datasets

labels (list) – labels specifying which DataObject values except for a tensor will be returned by the next() call

transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor

constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method

TypeError – raised if any of the parameters has an invalid type

NotADirectoryError – raised if the passed path does not point to any directory

This iterator does not support a parallelized resource loading.

always returns False

Returns a dataset subset using provided constraints

constraints (dict) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

a valid gemicai.data_iterators.PickledDicomoDataFolder object

TypeError – raised whenever constraints parameter is not a dict

Bases:

This class takes in a valid path to a .gemset dataset and iterates over it.
It’s constructor takes in the following parameters:

pickle_path (str) – a path to a valid .gemset file

labels (Optional[list]) – labels specifying which DataObject values except for a tensor will be returned by the next() call

transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor

constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method

TypeError – raised if any of the parameters has an invalid type

FileNotFoundError – raised whenever passed pickle_path does not point to any existing file

This iterator does not support a parallelized resource loading.

always returns False

Returns a dataset subset using provided constraints

constraints (dict) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

a valid gemicai.data_iterators.PickledDicomoDataSet object

TypeError – raised whenever constraints parameter is not a dict

Bases:

This class takes in a list of files as an input and iterates over them.
It’s constructor takes in the following parameters:

file_pool (list) – list of a valid file paths to .gemset datasets

labels (list) – labels specifying which DataObject values except for a tensor will be returned by the next() call

transform (Optional[any torchvision.transforms]) – optional transforms to be applied on the tensor

constraints (Optional[dict]) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

label_counter_type (gemicai.label_counter.GemicaiLabelCounter) – label counter used by the summarize method

TypeError – raised if any of the parameters has an invalid type

FileNotFoundError – raised if some path in the file_pool does not point to any existing file

This iterator does not support a parallelized resource loading.

always returns False

Returns a dataset subset using provided constraints

constraints (dict) – optional constraints that the DataObject has to fulfil in order to be returned by the
next() call, eg. ‘Modality’: ‘CT’ or ‘Modality’: [‘CT’, ‘MG’]

a valid gemicai.data_iterators.PickledDicomoFilePool object

TypeError – raised whenever constraints parameter is not a dict

gemicai.data_objects module

This module contains data objects used by the Gemicai’s iterators

Bases: abc.ABC

Every custom data object should extend this abstract base class and call it’s constructor.

This method should create and return a DataObject instance.

filename (Union[os.path, str]) – path to a valid file name

Call to this method should plot DataObject’s tensor to the screen.

Bases:

Gemicai’s default data object

Creates a DicomoObject from a specified file.

filename (Union[os.path, str]) – a valid dicom file path

labels (list) – labels which values will be taken from the Dicom object. The pixel_array field should not be
specified as it is one of the default fields fetched internally.

tensor_size (Optional[tuple]) – used to resize a tensor. If left unspecified it will try to use the current image size
otherwise it will use the specified values. Correct format ((int)x, (int)y)

DicomoObject instance

Returns a label value of a given label type.

item (str) – string with a objects label

value of a label or None if the object does not contain it

Checks whenever the object meets a certain type of criteria.

constraints (dict) – constraints to check against eg. ‘Modality’: ‘CT’

True if the object meets criteria, False otherwise

Prints labels and plots the tensor.

cmap (str) – color scheme

gemicai.dicom_utilities module

This module contains some utility functions that are used by the Gemicai in order to interface with dicom objects

Creates a Gemicai dataset from the data_origin (it should contain a valid dicom objects) and puts them in the
data_destination.

data_origin (Union[str, io.path]) – path to a folder containing dicom files

data_destination (Union[str, io.path]) – path to a destination where the gemsets will be outputted

relevant_labels (list) – specify which labels along with their values to extract from the dicom file and put into
gemicai.data_objects.DicomObject, eg. [‘Modality’] in this case DicomObject will contain a tensor and its
Modality

field_values (Optional[list]) – dataset will contain only objects which fulfil specified critieria,
eg. [(‘Modality’, [‘CT’, ‘MG’]), …] in this case dataset will contain only objects whose Modality is set to
CT or MG

objects_per_file (Optional[int]) – specifies how many objects one gemicai dataset should contain. A default value is 1000

pick_middle (bool) – specifies whenever instead of taking all images from the series only the middle one is taken.
This can be useful if someone is dealing with series spanning a multiple of dicom objects.

verbosity (int) – optional non-negative parameter, if set to one it will output how long it took to process all of
the data from data_origin

NotADirectoryError – raised if data_origin or data_destination does not point to an existing directory

TypeError – raised if any of the parameters has a wrong type or its value is out of the accepted bounds

Extracts an image from the dicom file and creates a tensor out of it

ds (pydicom.dataset.FileDataset) – dicom object to extract an image from

torch.Tensor

Loads in a given dicom file using a pydicom library

filename (Union[str, os.path]) – a path to the .dcm.gz or .dcm file

pydicom.dataset.FileDataset or pydicom.dicomdir.DicomDir

TypeError – raised if the file extension does not end with .dcm nor .gz

Plots image stored in a given dicom file. If a path given instead it will try to load a specified file first.

dcm (Union[str, pydicom.dataset.FileDataset]) – dicom object or a valid path to a dicom file

cmap (str) – color scheme

gemicai.label_counters module

This module contains label counters which are used by the data iterators in order to count distinct data
classes present in the dataset

Bases: abc.ABC

Every custom label counter should extend this abstract base class

This function is called whenever we have to count number of unique classes in a given input

labels (any) – in case of a user defined label counter it has to hold values to check against

Bases:

Gemicai’s default label counter implementation

This function checks if a given input is in it’s internal mapping if not it is added to it and it’s counter
is set to one, otherwise if it is already present then the counter is incremented by one.

labels (Union[list, str, pydicom.valuerep.IS]) – contains labels to count

gemicai.output_policies module

This module contains output policies. Such policy can be supplied as an optional parameter during model
training or evaluation in order to log, save, or print statistics related to the model’s performance.

Bases: abc.ABC

Every custom policy should extend this abstract base class.

Called after a model evaluation finishes if verbosity is set to 1.

total (int) – number of total objects model was evaluated on

correct (int) – number of correctly classified objects

acc (float) – overall accuracy of the model

Called after a model evaluation finishes if verbosity is equal or greater than 2.

classes (list) – list with class names on which model was evaluated

class_total (list) – list with a number of classes on which model was evaluated

class_correct (list) – list with a number of properly classified classes

This function is called every time an epoch ends, all of the important training statistics are taken as
an input.

epoch (int) – current training epoch epoch

running_loss (float) – current total running loss of the model

total (int) – number of the images that model has trained on

train_acc (str) – models accuracy on the provided train dataset

test_acc (str) – models accuracy on the optionally provided eval dataset

elapsed (str) – total time it took to run the epoch

eta (str) – an estimated time when training will end

Called once when training has finished.

start (datetime.datetime) – time when the training has started

now (datetime.datetime) – current time

Class implementing this method should make sure to specify how to handle a training header call.
This call happens once before training. It’s purpose is to beautify training_epoch_stats output by providing
some context into what categories of data the user is looking at.

Bases:

This policy allows to output training statistics to the console

Outputs model evaluation statistics to the console if verbosity is set to 1.

total (int) – number of total objects model was evaluated on

correct (int) – number of correctly classified objects

acc (float) – overall accuracy of the model

Outputs model evaluation statistics to the console if verbosity is equal or greater than 2.

classes (list) – list with class names on which model was evaluated

class_total (list) – list with a number of classes on which model was evaluated

class_correct (list) – list with a number of properly classified classes

an input and outputted to the console.

epoch (int) – current training epoch epoch

running_loss (float) – current total running loss of the model

total (int) – number of the images that model has trained on

train_acc (str) – models accuracy on the provided train dataset

test_acc (str) – models accuracy on the optionally provided eval dataset

elapsed (str) – total time it took to run the epoch

eta (str) – an estimated time when training will end

Outputs elapsed training time to the console.

start (datetime.datetime) – time when the training has started

now (datetime.datetime) – current time

Prints a training header to the console

Bases: ,

This output policy is a composition of ToConsole and ToExcelFile policies.

Called after a model evaluation finishes if verbosity is set to 1. Outputs the training statistics to
the console and to the specified excel file.

total (int) – number of total objects model was evaluated on

correct (int) – number of correctly classified objects

acc (float) – overall accuracy of the model

Called after a model evaluation finishes if verbosity is equal or greater than 2. Outputs the training
statistics to the console and to the specified excel file.

classes (list) – list with class names on which model was evaluated

class_total (list) – list with a number of classes on which model was evaluated

class_correct (list) – list with a number of properly classified classes

as an input and outputs them to the console and the specified excel file.

epoch (int) – current training epoch epoch

running_loss (float) – current total running loss of the model

total (int) – number of the images that model has trained on

train_acc (str) – models accuracy on the provided train dataset

test_acc (str) – models accuracy on the optionally provided eval dataset

elapsed (str) – total time it took to run the epoch

eta (str) – an estimated time when training will end

Called once when training has finished. Outputs elapsed time to the console and the specified excel file.

start (datetime.datetime) – time when the training has started

now (datetime.datetime) – current time

Outputs training header to the console and the specified excel file

Bases:

This policy allows to output training statistics to the excel file

Called after a model evaluation finishes if verbosity is set to 1. Outputs a training statistics to the
specified excel file.

total (int) – number of total objects model was evaluated on

correct (int) – number of correctly classified objects

acc (float) – overall accuracy of the model

Called after a model evaluation finishes if verbosity is equal or greater than 2.
Outputs a training statistics to the specified excel file.

classes (list) – list with class names on which model was evaluated

class_total (list) – list with a number of classes on which model was evaluated

class_correct (list) – list with a number of properly classified classes

This function writes each entry from data_list into a separate cell.

data_list (list) – list with data to be written

cells (list) – list with column names. Should have at least as many entries as data_list.

an input and outputted to the specified excel file.

epoch (int) – current training epoch epoch

running_loss (float) – current total running loss of the model

total (int) – number of the images that model has trained on

train_acc (str) – models accuracy on the provided train dataset

test_acc (str) – models accuracy on the optionally provided eval dataset

elapsed (str) – total time it took to run the epoch

eta (str) – an estimated time when training will end

Called once when training has finished, it outputs the elapsed time to the specified excel file.

start (datetime.datetime) – time when the training has started

now (datetime.datetime) – current time

Outputs a training header to the excel file