Supported Frameworks for AI Factsheet

class FrameworkSupportOptions

To view supported frameworks , use:

>>> client.FrameworkSupportNames.show()

To see only supported framework names , use:

>>> client.FrameworkSupportNames.get()

Set of Supported Frameworks for Auto logging. Manual log is not version dependent.

Available Options:

Name	Framework Name	Autolog	Manual Log	Version
scikit	sklearn	Y	Y	0.22.1 <= scikit-learn <= 1.1.2
Tensorflow	tensorflow	Y	Y	2.3.0 <= tensorflow <= 2.9.1
Keras	keras	Y	Y	2.3.0 <= keras <= 2.9.0
PySpark	pyspark	Y	Y	3.0.0 <= pyspark <= 3.4.0
Xgboost	xgboost	Y	Y	1.1.1 <= xgboost <= 1.6.1
LightGBM	lightgbm	Y	Y	2.3.1 <= lightgbm <= 3.3.2
PyTorch	pytorch	Y	Y	1.0.5 <= pytorch-lightning <= 1.7.1
Pycaret	pycaret	Y	N

Undoc-members:

Scikit learn

class FrameworkSupportSklearn

Current autolog support scope for Scikit learn.

Available Options:

Framework Name	Trigger Methods	Training Metrics	Parameters	Tags	Post Training Metrics	Search Estimators
sklearn	estimator.fit() estimator.fit_predict() estimator.fit_transform()	Classifier: precision score recall score f1 score accuracy score If the classifier has method predict_proba log loss roc auc score Regression: mean squared error root mean squared error mean absolute error r2 score	estimator.get_params(deep=True)	estimator class name(e.g. “LinearRegression”) fully qualified estimator class name(e.g. “sklearn.linear_model._base.LinearRegression”)	Scikit-learn metric APIs: model.score metric APIs defined in the sklearn.metrics module Note: metric key format is: {metric_name}[-{call_index}]_{dataset_name} if sklearn.metrics: metric_name is the metric function name if model.score, then metric_name is {model_class_name}_score If multiple calls are made to the same scikit-learn metric API each subsequent call adds a “call_index” (starting from 2) to the metric key	Meta estimator: Pipeline, GridSearchCV , RandomizedSearchCV It logs child runs with metrics for each set of explored parameters, as well as parameters for the best model and the best parameters (if available)

Note

• In notebook setting, if post training metrics need to be logged, make sure including all calculations in one cell with estimator fit() call. Metrics calculation not part of the same cell will not be logged automatically.

• Scikit-learn metric APIs invoked on derived objects do not log metrics

• If user define a scorer which is not based on metric APIs in sklearn.metrics, then then post training metric autologging for the scorer is invalid

PySpark

class FrameworkSupportSpark

Current autolog support scope for Spark.

Available Options:

Framework Name	Trigger Methods	Training Metrics	Parameters	Tags	Post Training Metrics	Search Estimators
pyspark	estimator.fit(), except for estimators (featurizers) under pyspark.ml.feature	Not Supported	estimator.params If a param value is also an Estimator then params in the the wrapped estimator will also be logged, the nested param key will be {estimator_uid}.{param_name}	estimator class name(e.g. “LinearRegression”) fully qualified estimator class name(e.g. “pyspark.ml.regression.LinearRegression”)	pyspark ML evaluators used under Evaluator.evaluate metric key format is: {metric_name}[-{call_index}]_{dataset_name} Metric name: Evaluator.getMetricName() If multiple calls are made to the same pyspark ML evaluator metric API each subsequent call adds a “call_index” (starting from 2) to the metric key	Meta estimator: Pipeline, CrossValidator, TrainValidationSplit, OneVsRest It logs child runs with metrics for each set of explored parameters, as well as parameters for the best model and the best parameters (if available)

Note

• In notebook setting, if post training metrics need to be logged, make sure including all calculations in one cell with estimator fit() call. Metrics calculation not part of the same cell will not be logged automatically.

• Facts client can not find run information for other objects derived from a given prediction result (e.g. by doing some transformation on the prediction result dataset)

Keras

class FrameworkSupportKeras

Current autolog support scope for Keras.

Available Options:

Framework Name	Trigger Methods	Training Metrics	Parameters
keras	estimator.fit()	Training loss, Validation loss, User specified metrics, Metricss related EarlyStopping callbacks: stopped_epoch, restored_epoch, restore_best_weight, last_epoch etc.	fit() or fit_generator() params, Optimizer name, Learning rate, Epsilon, Params related to EarlyStopping: min-delta, patience, baseline, restore_best_weights etc.

Note

• EarlyStopping Integration with Keras AutoLogging will detect if an EarlyStopping callback is used in a fit() or fit_generator() call, and if the restore_best_weights parameter is set to be True, then Factsheet will log the metrics associated with the restored model as a final, extra step. The epoch of the restored model will also be logged as the metric restored_epoch. This allows for easy comparison between the actual metrics of the restored model and the metrics of other models.If restore_best_weights is set to be False, then Factsheet will not log an additional step.

• Regardless of restore_best_weights, client will also log stopped_epoch, which indicates the epoch at which training stopped due to early stopping.

• If training does not end due to early stopping, then stopped_epoch will be logged as 0.

• Client will also log the parameters of the EarlyStopping callback, excluding mode and verbose

Tensorflow

class FrameworkSupportTensorflow

Current autolog support scope for Tensorflow.

Available Options:

Framework Name	Trigger Methods	Training Metrics	Parameters
tensorflow	estimator.fit()	Training loss, Validation loss, User specified metrics, Metricss related EarlyStopping callbacks: stopped_epoch, restored_epoch, restore_best_weight, last_epoch etc. TensorBoard metrics: average_loss, loss Tensorflow Core: tf.summary.scalar calls	fit() or fit_generator() params, Optimizer name, Learning rate, Epsilon, Params related to EarlyStopping: min-delta, patience, baseline, restore_best_weights etc. Tensorboard params: steps, max_steps

XGBoost

class FrameworkSupportXGB

Current autolog support scope for XGBoost.

Available Options:

Framework Name	Trigger Methods	Training Metrics	Parameters
xgboost	xgboost.train(), scikit-learn APIs.`fit()`	Metrics at each iteration (if evals specified), Metrics at best iteration (if early_stopping_rounds specified)	params specified in xgboost.train or fit()

LightGBM

class FrameworkSupportLGBM

Current autolog support scope for LightGBM.

Available Options:

Framework Name	Trigger Methods	Training Metrics	Parameters
lightgbm	lightgbm.train()	Metrics at each iteration (if evals specified), Metrics at best iteration (if early_stopping_rounds specified)	params specified in lightgbm.train

PyTorch

class FrameworkSupportPyTorch

Current autolog support scope for PyTorch.

Available Options:

Framework Name	Trigger Methods	Training Metrics	Parameters
pytorch	pytorch_lightning.Trainer() i.e., models that subclass pytorch_lightning.LightningModule	Training loss, Validation loss, average_test_accuracy, user defined metrics, Metricss related EarlyStopping callbacks: stopped_epoch, restored_epoch, restore_best_weight, last_epoch etc.	fit() parameters, optimizer name, learning rate, epsilon, Params related to EarlyStopping: min-delta, patience, baseline, restore_best_weights etc.

Note

• Only PyTorch Lightning models are supported.

• Vanilla PyTorch models that only subclass torch.nn.Module is not yet available.

Pycaret

Attention

• When using PyCaret, ensure that the experiment names in the Facts Client match those in the PyCaret `setup()` method. Consistency in experiment names is crucial for proper functionality.

Note

• PyCaret to function correctly, set `enable_push_framework` to true during client initialization.

• PyCaret can be invoked by `setup()` method and the `compare_models()` method. Ensure to pass `log_experiment=True` in the `setup()` method for proper logging.

• PyCaret is utilized with ‘autolog’ enabled. Manual logging is not recommended.