Version 0.15

Important

Trains is now ClearML.

Version 0.15.1

Trains

Features

  • Add direct plotly figure reporting (GitHub Issue #136).

  • Improve matplotlib integration (GitHub Issue #140).

  • Add Task.set_parent (GitHub Issue #144).

  • Add Task.wait_for_status (GitHub Issue #145).

  • Enhance framework save / load binding callbacks (pytorch/ignite Issue #1056).

  • Add TrainsJob.is_failed.

  • Add APIClient support using trains.backend_api.session.APIClient.

  • Add URL tracking when using StorageManager.get_local_copy (log remote URL instead of local file copy).

  • Improve and expand examples.

Bug Fixes

  • Fix pytorch_tensorboard.py example (GitHub Issue #148).

  • Fix ssh://user@domain@server.com/ in Git repository (GitHub Issue #149).

  • Fix do not call make_deterministic on import.

  • Fix initialization wizard (allow at most two verification retries, then print error) (GitHub trains-agent Issue #16).

  • Fix Google CoLab code/package detection.

  • Improve pytorch-ignite integration.

  • Fix TensorFlow version 2 and later histogram binding.

  • Fix Logger.tensorboard_single_series_per_graph.

  • Fix anonymous named models.

  • Fix incorrect entry point detection when called from Trains wrapper (e.g. TrainsLogger in Pytorch Ignite/Lightning).

Trains Server

Bug Fixes

  • Fix experiment clone dialog resets while typing (GitHub trains-server Issue 44).

  • Fix image-only plots are not displayed (GitHub Issue #140).

  • Fix last item in plots list is hidden.

  • Fix input model edit view contains a redundant save button.

  • Fix credentials popup shows incorrect URL for fileserver.

  • Fix experiment description does not allow multiple lines.

  • Fix tags filter (“No tags” selection, display only tags available for the project).

  • Fix cannot save empty model configuration.

  • Fix add tag dialog does not close after adding a tag.

  • Show active column filters for values even when values are unavailable in current project.

  • Fix project selection resets when cloning an experiment.

  • Fix several broken icons.

  • Fix cannot add tags in Results view.

  • Add warning when deleting a model/experiment tag.

  • New icons for experiment/task types.

Trains Agent

Features

  • Add Trains Agent Daemon and Services docker files.

Bug Fixes

Version 0.15.0

Trains

Features

  • Add automation support including hyper-parameters optimization (see example here)

  • Task.init auto_connect_arg_parser argument can accept a dictionary disabling specific keys from the argparser (Trains Slack channel thread)

  • Allow worker_id override using TRAINS_WORKER_NAME environment variable (Trains Slack channel thread)

  • Support layout configuration for plotly objects using extra_layout argument in all Logger reporting methods (GitHub Issue #136).

  • Add Task.execute_remotely to allow cloning and enqueuing a locally executed task (or stopping and re-enqueuing a remotely executed task) (GitHub Issue #128).

  • Add Parquet framework and model type

  • Support recursive model folder packaging

  • Add Task.get_reported_console_output and Task.get_reported_scalars to allow retrieval of reported output and scalar metrics

  • Add Task.add_requirements to force requirement package into “installed packages”

  • Improve task reuse responsiveness

  • Add raise_on_error (default False) argument to Model.get_local_copy and Artifact.get_local_copy (GitHub trains-agent Issue #17).

  • Support Task.get_task without project name (i.e. all projects)

  • Support using the file calling Task.init as the task’s script in case sys.argv doesn’t point to a git repository

  • Support detecting and remotely executing code running from a module (i.e. -m module)

  • Add callback for framework save / load binding for better integration with pytorch/ignite (GitHub pytorch/ignite Issue #1056)

  • Support new task types provided in Trains Server v0.15.0

  • Add automation and distributed examples

  • Upgrade default pip version to <20.2

Bug Fixes

  • Fix exact_match_regex in case of empty pattern (GitHub Issue #138).

  • Address deprecation warning and newer attrs versions in MetricsEventAdapter (GitHub Issue #134).

  • Fix issues with plotly support (Trains Slack channel thread and thread)

  • Fix default argument behavior to match argparse behavior

  • Fix OutputModel with task=None should use current task, if exists

  • Fix Task.get_task() to raise proper error on incorrect task_id

  • Fix Task.enqueue() to use an exact queue name match

  • Fix NaN, Inf and -Inf values display in reported table (not supported by JSON)

  • Limit max requirement size to 0.5MB

  • Fix issues with repository analysis

  • Fix StorageManager should only try to extract .zip files, Model should not auto extract package (GitHub trains-agent Issue #17).

Trains Server

Features

  • Add support for debug media (images, audio, video) (GitHub Issue #120).

  • Add support for previewing HTML debug samples.

  • Add experiment and model tag management (GitHub Issue #46).

  • Add support for more task types (see Trains v0.15.0 and Trains Agent v0.15.0 for related support and services mode).

  • Experiment comparison:

    • Add Last / Max (maximal) / Min (minimal) scalar values to parallel coordinates in experiment comparison (GitHub Issue #131).

    • Add deep comparison support to experiment comparision, including uncommitted changes by file with line diff and search of all text.

  • Enhance table columns:

    • Add dynamic column ordering to the experiments and models tables.

    • Add the User column to the experiments table.

    • Add the Description to experiments and models tables.

    • Add double click to table column resizers for column width reset.

    • Add column customization to the models table.

  • Add support for saving and sharing customized model table URLs by embedding table customizations in the URL.

  • Add a popup text editor for all text areas.

  • Move resource monitoring plots after the metrics plots in the experiment results SCALARS tab.

  • Add support for the clear_all parameter in tasks.reset to improve Trains SDK Task reuse.

  • Unify experiments table and comparison auto-refresh (switching between table and comparison auto refresh setting did not change).

  • Add a cancel button to all text searches.

  • Improve the experiment / info panel split to 75% / 25%.

  • Infrastructure:

    • Update pymongo and mongoengine to their latest versions.

    • Improve mongo performance by eliminating non-indexed queries.

    • Upgrade WebApp to Angular 9.

    • Use native javascript instead of date-fns.

  • Add survey invitation popup.

Bug Fixes

  • Fix init wizard, correctly display the input servers (GitHub Issue #19).

  • Improve docker host-mount support, use TRAINS_AGENT_DOCKER_HOST_MOUNT environment variable.

  • Support pip v20.1 local/http package reference in pip freeze.

  • Fix detached mode to correctly use cache folder slots.

  • Fix CUDA_VISIBLE_DEVICES should never be set to “all” (from the Trains Slack channel).

  • Do not monitor GPU when running with --cpu-only

  • Fix experiment comparison issues:

    • Scalar comparison type does not reflect displayed comparison (GitHub Issue #40).

    • Comparison animations overlapping other fields.

    • Cannot choose a metric after clearing it in experiment comparison parallel coordinates.

    • Adding a dimension without a value removes all lines in experiment comparison parallel coordinates.

  • Fix experiment view issues:

    • Custom URL gets erased when clicking on any tab on experiment.

    • All experiments re-fetched when navigating through tabs of an experiment.

    • Input and output models configuration box sizes differs in the experiments ARTIFACTS tab.

    • Changing experiment table / info panel split redraws scalars and plots.

    • Wrong download URL shown for S3 artifact links with special characters.

  • Fix experiment results view issues:

    • Browser zoom breaks labels (Ctrl + / Ctrl - / mouse wheel) in experiment results SCALARS tab (GitHub Issue #127).

    • Debug image name not shown if file cannot be loaded.

    • minio images with spaces in the path not displaying correctly

    • Plots should use the full panel space.

    • Plot relative time displaying incorrectly in experiment results SCALARS tab.

    • Wall time display in local time in experiment results SCALARS tab.

  • Fix experiment table issues:

    • Hyper parameters sorted as strings, not as numbers (GitHub Issue #124).

    • Prevent auto refresh reordering when the info panel is open.

    • Table resizing made the preferences wheel to jump over titles.

  • Fix experiment reset when both the input and output model are the same causes model deletion.

  • Fix experiment clone not working if the input model was deleted.

  • Fix missing trash icon (delete) if a custom column title is too long.

  • Fix display of aborted tasks in the recent experiments on the Homepage.

Trains Agent

Features

  • Add daemon Services Mode (daemon --services-mode), where the daemon spins a task in its own Docker container, and verifies start-up and shut-down. This allows multiple tasks to be launched simultaneously on the same machine (currently in CPU mode only), where each task service will register itself as a worker for the lifetime of the task.

  • Enhance build --docker mode.

    • Add --install-globally option to install required packages in the Docker’s system Python.

    • Add --entry-point option to allow automatic task cloning when running the Docker container.

  • Support PyTorch Nightly builds using the agent.torch_nightly configuration flag. If true, the agent looks for a nightly build when a stable torch wheel is not found.

  • Add environment variables support for git user/password

    • Using TRAINS_AGENT_GIT_USER / TRAINS_AGENT_GIT_PASS.

    • Pass git credentials to dockerized experiment execution.

  • Support running code from module (i.e. -m in execution entry point).

  • Add daemon --create-queue to automatically create a queue and use it, if the queue name doesn’t exist in the server.

  • Move --gpus and --cpu-only to worker args (used by daemon, execute and build).

Bug Fixes

  • Fix init wizard, correctly display the input servers (GitHub Issue #19).

  • Improve docker host-mount support, use TRAINS_AGENT_DOCKER_HOST_MOUNT environment variable.

  • Support pip v20.1 local/http package reference in pip freeze.

  • Fix detached mode to correctly use cache folder slots.

  • Fix CUDA_VISIBLE_DEVICES should never be set to “all” (from the Trains Slack channel).

  • Do not monitor GPU when running with --cpu-only