Version 0.14


Trains is now ClearML.

Version 0.14.3



  • Add the Logger.report_histogram method mode parameter. The values are group (the default), stack, and relative.

  • Add PEP610 support for the new pip version 20.1 and newer, Git reference feature (see pypa/pip#609) (GitHub Issue #62).

  • Better support for PyTorch Ignite integration.

  • Support # and ? in uploaded image file names.

Bug Fixes

  • Support ArgumentParser.parse_arg use before initializing a Task when auto_connect_arg_parser=False.

  • Fix upload_object_via_stream in the Azure storage driver.

  • Fix the Task.get_tasks method return type.

Version 0.14.2



  • Add media (audio) support for both Logger and TensorBoard binding using Logger.report_media (GitHub Issue #120).

  • Add autokeras example (see here) (GitHub Issue #125).

  • Add caching of extracted zip artifacts.

  • Add Task.get_tasks filtering support.

  • Add

  • Add nicer stdout log flush.

  • Add text logging to TF V1 example.

  • Improve SSL retries and error handling.

  • Improve shutdown behavior in case of a user abort (Ctrl-C).

  • Improve repository and module detection:

    • Improve resource monitor.

    • Report memory usage for process (and sub-processes).

    • Report GPU memory usage for process (and sub-processes).

    • Stability

  • Improve Jupyter support:

    • Make sure trains is included in Jupyter requirements.

    • Ignore IPython directives in converted Python script (like % and ! lines).

  • Update Pytorch/TensorboardX examples.

Bug Fixes

  • Fix renaming/deleting model file right after saving will break asynchronous upload (GitHub Issue #123).

  • Fix shutdown behavior:

    • Logger flush delay and termination.

    • Repository/package detection termination.

    • Threads termination.

  • Fix text encoding utf-8 and pr_curve broken in Tensorboard support.

  • Fix single log request exceeds max packet size.

  • Fix upload debug assets with series / title including , or / did not add iteration index to the filename.

  • Fix scikit-learn module detection.

  • Use UID in message if user name cannot be retrieved (from the Trains Slack channel).

  • Fix cast in Tensorflow v2 example.

Trains Server

User survey

  • Add survey invitation popup.

Bug fixes

  • Fix misaligned titles in experiment results, scalars and plots (RESULTS tab, SCALARS and PLOTS sub-tabs).

  • Fix parallel coordinates comparison in comparing experiments. Adding dimension with no values removes all lines.

  • Fix resizing the experiments tables window. The preferences wheel jumped over the titles.

  • Fix the browser zoom (CTRL +, CTRL -, and mouse wheel) in experiment results, scalars (RESULTS tab, SCALARS sub-tab). It broke labels (GitHub Issue #127).

Version 0.14.1


Features and Bug Fixes

  • Add OpenMPI/Slurm support, support TRAINS_FORCE_MASTER_NODE environment variable to allow forcefully specifying a node is master (not based on the MPI Rank).

  • Add Task.get_models and Task.models for retrieving stored models for previously executed tasks

  • Support multiple EventWriter in TensorFlow eager mode (TF 2.0+) (GitHub Issue #26).

  • Support reusing models.

  • Fix TensorFlow not a number (NaN) and infinity (Inf) values support (GitHub Issue #118).

  • Fix joblib binding.

  • Fix metrics/files/models upload on shutdown.

  • Fix support for Task init/close multiple times.

  • Update example with Logger.report_table example.

  • Deprecate Task.set_model_config, Task.get_model_config_text and Task.get_model_config_dict.

Trains Server

Features and Bug Fixes

  • Support default Trains Server Kubernetes Helm ports in the Trains Web-App (UI).

  • Fix issue switching scalar plots between linear and logarithmic views (GitHub Issue #75).

  • Do not reset view filtering when switching tabs in experiment view.

Trains Agent

Features and Bug Fixes

  • Add daemon detached mode (--detached, -d) that runs the agent as daemon in the background and returns immediately.

  • Auto mount ~/.git-credentials into a Docker container, if the file exists.

  • Add TRAINS_AGENT_EXTRA_PYTHON_PATH environment variable to allow the adding of an additional Python path during experiment execution (helpful when using extra un-tracked modules).

  • Fix “run as user” feature (using TRAINS_AGENT_EXEC_USER environment variable).

  • Fix PyTorch support to ignore minor versions when looking for package to install/download.

  • Fix experiment execution output handling.

Version 0.14.0


Features and Bug Fixes

  • Add support for reporting tables (pandas.DataFrame, CSV files, URL to CSV file) using the Logger.report_table method.

  • Add support for setting a Task’s initial iteration to allow continuing previous runs using the Task.set_initial_iteration and Task.get_initial_iteration methods.

  • Add multiprocess task support (GitHub Issue #96).

  • Add trains-init support for config file override using the TRAINS_CONFIG_FILE environment variable or --file command line argument (GitHub Issue #105).

  • Add initial slurm support.

  • Allow disabling repository detection when calling Task.init.

  • Add Jupyter Notebook server execution example.

  • Update examples.

  • Improve warning messages for storage errors and missing storage packages.

    • Fix HTTP link quoting in stored links.

Trains Server

Features and Bug Fixes

  • Add debug image viewing for an unlimited number of iterations, filtering by metric, and comparing debug images for different iterations.

  • Add debug image view zoom, fit to screen, resize, hover coordinates (GitHub Issue #68).

  • Add switching scalar plots between linear and logarithmic views (GitHub Issue #75).

  • Add scalar minimums and maximums to comparing experiments.

  • Add opening HTML artifact files in the browser.

  • Add Web-App notification for version updates.

  • Add toggle button to switch between full screen and standard views in the RESULTS tab.

  • Add a load more button to the global search results.

  • Add automatic detection of debug video type from the file extension.

  • Add Task.get_parameters_as_dict and Task.set_parameters_as_dict.

  • Allow editing model description.

  • Allow editing experiment uncommitted changes and installed packages separately.

  • Allow deleting an input model from an experiment whose status is Draft.

  • Fix dialog elements which are now accessible in non-100% screen scale. (GitHub Issue #89).

  • Fix custom columns in the experiments table not properly loaded from browser store (GitHub Issue #106).

  • Fix issue with showing 2D graphs containing sub-plots (trains #slack)

  • Fix issues with vertical scrolling of experiment logs.

  • Fix issue with image plots sometimes not visible in experiments comparison.

Trains Agent

Features and Bug Fixes

  • Add support for trains-agent execute --id <experiment-id> --docker that allows executing a specific experiment inside a docker container.

  • Add support for trains-agent execute --id <template-experiment-id> --clone that clones the provided experiment and executes the cloned experiment.

  • Add support for APIClient.models.delete to allow programmatically deleting a model (GitHub trains-server Issue 32.

  • Add daemon support for passing storage-related OS environment variables to experiments executed inside a Docker container (supported by trains>=0.13.3):




  • Fix Git checkout with submodules (GitHub Issue #112).

  • Docker image in command line takes precedence over the Docker image specified in an experiment.