Link Search Menu Expand Document

Analyzer

The Analyzer component provides multiple analyses and metrics in order to evaluate the performances of the model.
It is possible to evaluate the model on the whole data set or only on a subset, splitting the evaluation for each category or for each meta-annotation value.


AnalyzerClassification

The AnalyzerClassification class can be used to perform diagnostics for classification models.

Parameters

classifier_name

str
Name of the classifier. It is used as folder to save results.
dataset

DatasetClassification
Dataset used to perform the analysis.
result_saving_path

str, optional
Path used to save results.
(default is './results/')
use_normalization

bool, optional
Indicates whether normalisation should be used.
(default is False)
norm_factor_categories

float, optional
Normalisation factor for the categories. If not specified, the default value is 1/number_of_categories.
(default is None)
norm_factors_properties

list of pair, optional
Normalization factor for the properties. Each pair (property_name, value) specifies the normalisation factor to be applied to a specific property. If not specified, for each property the default value is 1/number_of_property_values.
(default is None)
conf_thresh

float, optional
Confidence threshold. All the predictions with a confidence value less than the threshold are ignored.
If not specified, for single-label classification problems, the default value is 0.
If not specified, for binary and multi-label classification problems the default value is 0.5.
(default is None)
metric

Metrics, optional
The evaluation metric that will be used as default.
(default is Metrics.F1_SCORE)
save_graphs_as_png

bool, optional
Indicates whether plots should be saved as .png images.
(default is True)

Example

from odin.classes import AnalyzerClassification

# N.B. If different models proposals have been loaded to the dataset, the
#      classifier name must be the same of one of the models names provided
#      with the predictions when my_classification_dataset has been created
my_analyzer = AnalyzerClassification("my_classifier_name", my_classification_dataset)

AnalyzerLocalization

The AnalyzerLocalization class can be used to perform diagnostics for localization models, such as object detection and instance segmentation.

Parameters

detector_name

str
Name of the detector. It is used as folder to save results.
dataset

DatasetLocalization
Dataset used to perform the analysis.
result_saving_path

str, optional
Path used to save results.
(default is './results/')
use_normalization

bool, optional
Indicates whether normalisation should be used.
(default is False)
norm_factor_categories

float, optional
Normalisation factor for the categories. If not specified, the default value is 1/number_of_categories.
(default is None)
norm_factors_properties

list of pair, optional
Normalization factor for the properties. Each pair (property_name, value) specifies the normalisation factor to be applied to a specific property. If not specified, for each property the default value is 1/number_of_property_values.
(default is None)
iou

float, optional
Intersection Over Union threshold. All the predictions with a iou value less than the threshold are considered False Positives. If not specified, the default value is 0.5.
(default is None)
iou_weak

float, optional
Intersection Over Union weak threshold. Used for the identification of the localization errors. If not specified, the default value is 0.1.
(default is None)
conf_thresh

float, optional
Confidence threshold. All the predictions with a confidence value less than the threshold are ignored.
If not specified, the default value is 0.5.
(default is None)
metric

Metrics, optional
The evaluation metric that will be used as default.
(default is Metrics.Metrics.AVERAGE_PRECISION_SCORE)
save_graphs_as_png

bool, optional
Indicates whether plots should be saved as .png images.
(default is True)

Example

from odin.classes import AnalyzerLocalization

# N.B. If different models proposals have been loaded to the dataset, the
#      detector name must be the same of one of the models names provided
#      with the predictions when my_localization_dataset has been created
my_analyzer = AnalyzerLocalization("my_detector_name", my_localization_dataset)

AnalyzerCAMs

The AnalyzerCAMs class can be used to diagnose Class Activation Maps generated by classification models.

Parameters

classifier_name

str
Name of the classifier. It is used as folder to save results.
dataset

DatasetCAMs
Dataset used to perform the analysis.
cam_thresh

float, optional
CAM confidence threshold. All the pixels with a confidence value less than the threshold are ignored.
(default is 0.5)
cam_coverage_thresh

float, optional
CAM coverage threshold. All the CAMs with a iou less than the threshold do not cover the annotation.
(default is 0.5)
result_saving_path

str, optional
Path used to save results.
(default is './results/')
use_normalization

bool, optional
Indicates whether normalisation should be used.
(default is False)
norm_factor_categories

float, optional
Normalisation factor for the categories. If not specified, the default value is 1/number_of_categories.
(default is None)
norm_factors_properties

list of pair, optional
Normalization factor for the properties. Each pair (property_name, value) specifies the normalisation factor to be applied to a specific property. If not specified, for each property the default value is 1/number_of_property_values.
(default is None)
conf_thresh

float, optional
Confidence threshold. All the predictions with a confidence value less than the threshold are ignored.
If not specified, the default value is 0.5.
(default is None)
metric

Metrics, optional
The evaluation metric that will be used as default.
(default is Metrics.Metrics.CAM_GLOBAL_IOU)
save_graphs_as_png

bool, optional
Indicates whether plots should be saved as .png images.
(default is True)

Example

from odin.classes import AnalyzerCAMs

# N.B. If different models CAMs have been loaded to the dataset, the
#      classifier name must be the same of one of the models names provided
#      with the CAMs when my_cams_dataset has been created
my_analyzer = AnalyzerCAMs("my_classifier_name", my_cams_dataset)