Version 0.15¶
Important
Trains is now ClearML.
Version 0.15.1¶
Trains¶
Features
Add direct plotly figure reporting (GitHub Issue #136).
Improve matplotlib integration (GitHub Issue #140).
Add
Task.set_parent
(GitHub Issue #144).Add
Task.wait_for_status
(GitHub Issue #145).Enhance framework
save
/load
binding callbacks (pytorch/ignite Issue #1056).Add
TrainsJob.is_failed
.Add APIClient support using
trains.backend_api.session.APIClient
.Add URL tracking when using
StorageManager.get_local_copy
(log remote URL instead of local file copy).Improve and expand examples.
Bug Fixes
Fix
pytorch_tensorboard.py
example (GitHub Issue #148).Fix
ssh://user@domain@server.com/
in Git repository (GitHub Issue #149).Fix do not call
make_deterministic
on import.Fix initialization wizard (allow at most two verification retries, then print error) (GitHub trains-agent Issue #16).
Fix Google CoLab code/package detection.
Improve pytorch-ignite integration.
Fix TensorFlow version 2 and later histogram binding.
Fix
Logger.tensorboard_single_series_per_graph
.Fix anonymous named models.
Fix incorrect entry point detection when called from Trains wrapper (e.g.
TrainsLogger
in Pytorch Ignite/Lightning).
Trains Server¶
Bug Fixes
Fix experiment clone dialog resets while typing (GitHub trains-server Issue 44).
Fix image-only plots are not displayed (GitHub Issue #140).
Fix last item in plots list is hidden.
Fix input model edit view contains a redundant save button.
Fix credentials popup shows incorrect URL for fileserver.
Fix experiment description does not allow multiple lines.
Fix tags filter (“No tags” selection, display only tags available for the project).
Fix cannot save empty model configuration.
Fix add tag dialog does not close after adding a tag.
Show active column filters for values even when values are unavailable in current project.
Fix project selection resets when cloning an experiment.
Fix several broken icons.
Fix cannot add tags in Results view.
Add warning when deleting a model/experiment tag.
New icons for experiment/task types.
Trains Agent¶
Features
Add Trains Agent Daemon and Services docker files.
Bug Fixes
Fix initialization wizard (allow at most two verification retries, then print error).
Add warning on
--gpus
with no detected CUDA version (GitHub trains-agent Issue 24).Add
agent.force_git_ssh_protocol
configuration option to force all Git links tossh://
(GitHub trains-agent Issue 16).Add Git user/pass permission into pip package installation from Git repository (GitHub trains-agent Issue 22).
Version 0.15.0¶
Trains¶
Features
Add automation support including hyper-parameters optimization (see example here)
Task.init
auto_connect_arg_parser
argument can accept a dictionary disabling specific keys from the argparser (Trains Slack channel thread)Allow
worker_id
override usingTRAINS_WORKER_NAME
environment variable (Trains Slack channel thread)Support layout configuration for plotly objects using
extra_layout
argument in allLogger
reporting methods (GitHub Issue #136).Add
Task.execute_remotely
to allow cloning and enqueuing a locally executed task (or stopping and re-enqueuing a remotely executed task) (GitHub Issue #128).Add Parquet framework and model type
Support recursive model folder packaging
Add
Task.get_reported_console_output
andTask.get_reported_scalars
to allow retrieval of reported output and scalar metricsAdd
Task.add_requirements
to force requirement package into “installed packages”Improve task reuse responsiveness
Add
raise_on_error
(default False) argument toModel.get_local_copy
andArtifact.get_local_copy
(GitHub trains-agent Issue #17).Support
Task.get_task
without project name (i.e. all projects)Support using the file calling
Task.init
as the task’s script in case sys.argv doesn’t point to a git repositorySupport detecting and remotely executing code running from a module (i.e.
-m module
)Add callback for framework
save
/load
binding for better integration with pytorch/ignite (GitHub pytorch/ignite Issue #1056)Support new task types provided in Trains Server v0.15.0
Add automation and distributed examples
Upgrade default pip version to
<20.2
Bug Fixes
Fix
exact_match_regex
in case of empty pattern (GitHub Issue #138).Address deprecation warning and newer
attrs
versions in MetricsEventAdapter (GitHub Issue #134).Fix issues with plotly support (Trains Slack channel thread and thread)
Fix default argument behavior to match argparse behavior
Fix
OutputModel
withtask=None
should use current task, if existsFix
Task.get_task()
to raise proper error on incorrecttask_id
Fix
Task.enqueue()
to use an exact queue name matchFix
NaN
,Inf
and-Inf
values display in reported table (not supported by JSON)Limit max requirement size to 0.5MB
Fix issues with repository analysis
Fix
StorageManager
should only try to extract .zip files,Model
should not auto extract package (GitHub trains-agent Issue #17).
Trains Server¶
Features
Add support for debug media (images, audio, video) (GitHub Issue #120).
Add support for previewing HTML debug samples.
Add experiment and model tag management (GitHub Issue #46).
Add support for more task types (see Trains v0.15.0 and Trains Agent v0.15.0 for related support and services mode).
Experiment comparison:
Add Last / Max (maximal) / Min (minimal) scalar values to parallel coordinates in experiment comparison (GitHub Issue #131).
Add deep comparison support to experiment comparision, including uncommitted changes by file with line diff and search of all text.
Enhance table columns:
Add dynamic column ordering to the experiments and models tables.
Add the User column to the experiments table.
Add the Description to experiments and models tables.
Add double click to table column resizers for column width reset.
Add column customization to the models table.
Add support for saving and sharing customized model table URLs by embedding table customizations in the URL.
Add a popup text editor for all text areas.
Move resource monitoring plots after the metrics plots in the experiment results SCALARS tab.
Add support for the clear_all parameter in tasks.reset to improve Trains SDK Task reuse.
Unify experiments table and comparison auto-refresh (switching between table and comparison auto refresh setting did not change).
Add a cancel button to all text searches.
Improve the experiment / info panel split to 75% / 25%.
Infrastructure:
Update pymongo and mongoengine to their latest versions.
Improve mongo performance by eliminating non-indexed queries.
Upgrade WebApp to Angular 9.
Use native javascript instead of date-fns.
Add survey invitation popup.
Bug Fixes
Fix init wizard, correctly display the input servers (GitHub Issue #19).
Improve docker host-mount support, use
TRAINS_AGENT_DOCKER_HOST_MOUNT
environment variable.Support pip v20.1 local/http package reference in pip freeze.
Fix detached mode to correctly use cache folder slots.
Fix
CUDA_VISIBLE_DEVICES
should never be set to “all” (from the Trains Slack channel).Do not monitor GPU when running with
--cpu-only
Fix experiment comparison issues:
Scalar comparison type does not reflect displayed comparison (GitHub Issue #40).
Comparison animations overlapping other fields.
Cannot choose a metric after clearing it in experiment comparison parallel coordinates.
Adding a dimension without a value removes all lines in experiment comparison parallel coordinates.
Fix experiment view issues:
Custom URL gets erased when clicking on any tab on experiment.
All experiments re-fetched when navigating through tabs of an experiment.
Input and output models configuration box sizes differs in the experiments ARTIFACTS tab.
Changing experiment table / info panel split redraws scalars and plots.
Wrong download URL shown for S3 artifact links with special characters.
Fix experiment results view issues:
Browser zoom breaks labels (Ctrl + / Ctrl - / mouse wheel) in experiment results SCALARS tab (GitHub Issue #127).
Debug image name not shown if file cannot be loaded.
minio images with spaces in the path not displaying correctly
Plots should use the full panel space.
Plot relative time displaying incorrectly in experiment results SCALARS tab.
Wall time display in local time in experiment results SCALARS tab.
Fix experiment table issues:
Hyper parameters sorted as strings, not as numbers (GitHub Issue #124).
Prevent auto refresh reordering when the info panel is open.
Table resizing made the preferences wheel to jump over titles.
Fix experiment reset when both the input and output model are the same causes model deletion.
Fix experiment clone not working if the input model was deleted.
Fix missing trash icon (delete) if a custom column title is too long.
Fix display of aborted tasks in the recent experiments on the Homepage.
Trains Agent¶
Features
Add daemon Services Mode (
daemon --services-mode
), where the daemon spins a task in its own Docker container, and verifies start-up and shut-down. This allows multiple tasks to be launched simultaneously on the same machine (currently in CPU mode only), where each task service will register itself as a worker for the lifetime of the task.Enhance
build --docker
mode.Add
--install-globally
option to install required packages in the Docker’s system Python.Add
--entry-point option
to allow automatic task cloning when running the Docker container.
Support PyTorch Nightly builds using the
agent.torch_nightly
configuration flag. If true, the agent looks for a nightly build when a stable torch wheel is not found.Add environment variables support for git user/password
Using
TRAINS_AGENT_GIT_USER
/TRAINS_AGENT_GIT_PASS
.Pass git credentials to dockerized experiment execution.
Support running code from module (i.e. -m in execution entry point).
Add
daemon --create-queue
to automatically create a queue and use it, if the queue name doesn’t exist in the server.Move
--gpus
and--cpu-only
to worker args (used bydaemon
,execute
andbuild
).
Bug Fixes
Fix init wizard, correctly display the input servers (GitHub Issue #19).
Improve docker host-mount support, use
TRAINS_AGENT_DOCKER_HOST_MOUNT
environment variable.Support pip v20.1 local/http package reference in pip freeze.
Fix detached mode to correctly use cache folder slots.
Fix
CUDA_VISIBLE_DEVICES
should never be set to “all” (from the Trains Slack channel).Do not monitor GPU when running with
--cpu-only