Version 0.16

Version 0.16.1

Trains

Features

  • Enhance HyperParameter optimizer.

Bug Fixes

  • Fix typing dependency for Python<3.5 (GitHub trains Issue 184).
  • Fix git+https requirements handling, resolve top_level.txt package name (kerastuner from git was not detected).
  • Fix Task.get_reported_console_output() for new Trains Server API v2.9.
  • Fix cache handling for different partitions/drives/devices.
  • Disable offline mode when running remotely (i.e. executed by Trains Agent).
  • Fix artifact upload to only use file stream when not uploading a locally stored file (multipart upload is not supported on stream upload) (GitHub trains Issue 189).
  • Fix double-escaped model design text when connecting OutputModel.

Trains Server

IMPORTANT: Upgrading to this version requires a manual data migration.

Bug Fixes

  • Fix model page issue causing N/A to show after switching tabs (Trains Slack channel thread).
  • Removed experiments comparison limit (only 10 were allowed). Limit is now 100, configurable using services.tasks.multi_task_histogram_limit. (Trains Slack channel thread).
  • Fix scalar plots sometimes not calculated by the server in lower iteration values (Trains Slack channel thread).
  • Fix error while retrieving experiment log when only a few lines were reported (GitHub trains-server Issue 59).
  • Update Fixed User full-name on restart (Trains Slack channel thread).
  • Fix project ordering issue.
  • When loading plots, display a spinner and don't show "no data".
  • Improve logging to provide more coherent ElasticSearch connection status in server log.

Version 0.16.0

Trains

Features

  • Add continuing of previously executed experiments. Add Task.init() argument continue_last_task to continue a previously used Task (GitHub Issue #160).
  • Allow Task editing/creation from code. Task.export_task/import_task/update_task() (GitHub Issue #128).
  • Add offline mode. Use Task.set_offline() and Task.import_offline_session()
  • Support setting offline mode via TRAINS_OFFLINE_MODE=1 environment variable.
  • Support setting offline API version via TRAINS_OFFLINE_MODE=2.9 environment variable.
  • Automatically pickle all objects when uploading as artifacts, task.upload_artifact() argument auto_pickle=True (GitHub Issue #153).
  • Add multiple sections/groups support for Task hyper-parameters using Task.connect().
  • Add multiple configurations (files) using Task.connect_configuration.
  • Allow enabling OS environment logging using the sdk.development.log_os_environments configuration parameter (complements the TRAINS_LOG_ENVIRONMENT environment variable).
  • Add Optuna support for hyper-parameter optimization controller. OptimizerOptuna is now the default optimizer.
  • Add initial Keras-Tuner support (GitHub Issue keras-team/keras-tuner #334).
  • Add automatic FastAI logging. It is disabled if tensorboard is loaded (assuming TensorBoradLogger will be used).
  • Support Tensorboard text logging (add_text()) as debug samples (.txt files), instead of as console output.
  • Allow for more standard confusion matrix reporting. Logger.report_confusion_matrix() argument yaxis_reversed (flips the confusion matrix if True, default False) (GitHub Issue #165).
  • Add support for Trains Server 0.16.0 (API v2.9 support).
  • Allow disabling Trains update message from the log using the TRAINS_SUPPRESS_UPDATE_MESSAGE environment variable (GitHub Issue #157).
  • Add AWS EC2 Auto-Scaler service wizard and Service.
  • Improved and updated examples
  • Add Keras Tuner CIFAR10 example.
  • Add FastAI example.
  • Update PyTorch Jupyter notebook examples (GitHub Issue #150).
  • Support global requirements detection using pip freeze (set sdk.development.detect_with_pip_freeze configuration in trains.conf).
  • Add Task.get_projects() to get all projects in the system, sorted by last update time.

Bug Fixes

  • Fix UTC to time stamp in comment (GitHub Issue #152).
  • Fix and enhance GPU monitoring
  • Fix GPU stats on Windows machines (GitHub Issue #177).
  • More robust GPU monitoring (GitHub Issue #170).
  • Fix filename too long bug (GitHub trains-server Issue #49).
  • Fix TensorFlow image logging to allow images with no width/height/color metadata (GitHub Issue #182).
  • Fix multiprocessing Pool throw exception in pool hangs execution. Call original signal handler and re-flush stdout.
  • Fix plotly support for matplotlib 3.3.
  • Add Python 2.7 support for get_current_thread_id().
  • Update examples requirements.
  • Fix and improve signal handling.
  • Fix Tensorboard 2D convolution histogram, improve histogram accuracy on very small histograms.
  • Fix auto logging multiple argparse calls before Task.init().
  • Limit experiment Git diff logging to 500Kb. If larger than 500Kb, diff section will contain a warning and entire diff will be uploaded as an artifact named auxiliary_git_dif.
  • Fix requirements detection
  • Fix Trains installed from git+.
  • Fix when Trains is not directly imported.
  • Fix multiple -e packages were not detected (only the first one).
  • Fix running with Trains in PYTHONPATH resulted in double entry of trains.
  • Fix Task.set_base_docker() on main task to do nothing when running remotely.

Trains Server

IMPORTANT: Upgrading to this version requires a manual data migration.

Features

  • Add experiment hyperparameter grouping.
    • HYPER PARAMETERS tab renamed to CONFIGURATIONS.
    • CONFIGURATIONS tab contains the sections USER PROPERTIES, HYPER PARAMETERS, CONFIGURATION OBJECTS
    • Add user properties group. Key-value pairs always editable (USER PROPERTIES section).
    • Add command line options group * argparse and older experiments parameters (CONFIGURATIONS/HYPER PARAMETERS/Args).
    • Add TensorFlow definitions group (CONFIGURATIONS/HYPER PARAMETERS/TF_DEFINE).
    • Add environment variables group (CONFIGURATIONS/HYPER PARAMETERS/Environment).
  • Improve experiment model configuration.
    • Model design is in the ARTIFACTS tab.
    • Experiment model description is in the CONFIGURATIONS OBJECTS section in the CONFIGURATIONS tab.
  • Improve experiment comparison:
    • In hyperparameter parallel coordinate comparison, hover over an experiment name to highlight it on plot (GitHub Issue #53).
    • Remove fields providing no additional information from comparison.
  • Improve the model framework filter. Filter contains only frameworks used by models in the project.
  • Add configurable Trains services examples.
  • Add support for text debug samples in the DEBUG SAMPLES section in the RESULTS tab.
  • Add legend on/off toggle control for every plot.
  • Add clear button for text areas (GitHub trains-server Issue #42).
  • Reinstate the bottom bar Archive button.
  • Add Trains community links to left bar.
  • Add Hi-DPI display support.
  • Add debug.ping endpoint for simple health monitoring.
  • Add support for field exclusion in *.get_all endpoints.
  • Move to ElasticSearch 7. Requires manual data migration

Bug Fixes

  • Auto-fit column width on column resize double click.
  • Allow top-bar search if fewer than three characters are entered, and Enter is pressed.

Trains Agent

Features

  • Add agent.docker_init_bash_script configuration section to allow finer control over docker startup script.
  • Changed default docker image from nvidia/cuda to nvidia/cuda:10.1-runtime-ubuntu18.04 to support cudnn frameworks (e.g. TF).
  • Improve support for dockers with preinstalled conda environment.
  • Improve trains-agent-docker spinning.
  • Add daemon --order-fairness for round-robin queue pulling.

  • Add daemon --stop to terminate a running agent (assuming other arguments are the same)

    • If no additional arguments, Agents are terminated in lexicographical order.

  • Support cleanup of all log files on termination unless executed with --debug.

  • Add error message when Trains API Server is not accessible on startup.

Bug Fixes

  • Fix GPU Windows monitoring support (GitHub Issue #177).
  • Fix .git-credentials and .gitconfig mapping into docker.
  • Fix non-root docker image usage.
  • Fix docker to use UTF-8 encoding, so prints won't break it.
  • Fix --debug to set all loggers to DEBUG.
  • Fix task status change to queued should never happen during Task runtime.
  • Fix requirement_parser to support package @ git+http lines.
  • Fix GIT user/password in requirements and support for -e git+http lines.
  • Fix configuration wizard to generate trains.conf matching latest Trains definitions.