Trace and Customize Training Runs

Capture additional metrics, parameters, and tags for an experiment (identified by run_id ) before saving the model.

Note

  • The following utilities are subject to local storage which the facts client uses as an intermediary step. When using a Watson Studio notebook environment, reloading notebooks sessions might cause exceptions to local storage. In such cases, the utilities might not function as expected, but the exceptions do not impact the results saved to factsheet. Users will still be able to see results through Factsheets UI.

Experiments

class Experiments(root_directory=None)

Bases: object

Utility to explore current experiments.

list_experiments(max_results: int = 100) → DataFrame

List all active experiments.

Returns:

DataFrame object.

Return type:

Pandas.DataFrame

A way you might use me is:

>>> client.experiments.list_experiments()
get_current_experiment_id()

Shows current experiment id.

Returns:

str

A way you might use me is:

>>> client.experiments.get_current_experiment_id()

Runs

class Runs(root_directory=None)

Bases: object

Utilities to explore runs within any experiment.

list_runs_by_experiment(experiment_id: str, order_by: List[str] | None = None) → DataFrame

List all runs under any experiment

Parameters:

  • experiment_id (str) – ID of the experiment.

  • order_by – List of order_by clauses. Currently supported values are metric.key, parameter.key, tag.key.For example, order_by=["tag.release ASC", "metric.training_score DESC"]

Returns:

DataFrame object that satisfy the search expressions.

Return type:

Pandas.DataFrame

A way you might use me is:

>>> client.runs.list_runs_by_experiment("1")

>>> client.runs.list_runs_by_experiment("1", order_by=["metric.training_score DESC"]))
get_current_run_id()

Shows current active run id.

Returns:

str

A way you might use me is:

>>> client.runs.get_current_run_id()
log_metric(run_id: str, key: str, value: float, step: int | None = None) → None

Log a metric against the run ID.

Parameters:

  • run_id (str) – The unique id for run.

  • key (str) – Metric name.

  • value (float) – Metric value (float).

  • step (int) – Integer training step (iteration) at which was the metric calculated. Defaults to 0.

Returns:

None

A way you might use me is:

>>> client.runs.log_metric(run_id, "mae", .77)
log_metrics(run_id: str, metrics: Dict[str, float], step: int | None = None) → None

Log multiple metrics for the given run.

Parameters:

  • run_id (str) – The unique id for run.

  • metrics (dict) – Dictionary of metric_name: String -> value: Float.

  • step (int) – Integer training step (iteration) at which was the metric calculated. Defaults to 0.

Returns:

None

A way you might use me is:

>>> client.runs.log_metrics(run_id, {"mse": 2000.00, "rmse": 50.00})
log_param(run_id: str, key: str, value: Any) → None

Log a param against the run ID.

Parameters:

  • run_id (str) – The unique id for run.

  • key (str) – Param name.

  • value – Param value.Value is converted to a string.

Returns:

None

A way you might use me is:

>>> client.runs.log_param(run_id, "c", 1)
log_params(run_id, params: Dict[str, Any]) → None

Log multiple params for the given run.

Parameters:

  • run_id (str) – The unique id for run.

  • params (dict) – Dictionary of String -> value: (String, but will be string-ified if not)

Returns:

None

A way you might use me is:

>>> client.runs.log_params(run_id, {"n_estimators": 3, "random_state": 42})
set_tags(run_id: str, tags: Dict[str, Any]) → None

Log multiple tags for the given run.

Parameters:

  • run_id (str) – The unique id for run.

  • tags (dict) – Dictionary of tags names: String -> value: (String, but will be string-ified if not)

Returns:

None

A way you might use me is:

>>> client.runs.set_tags(run_id, {"engineering": "ML Platform",
"release.candidate": "RC1"})