Yasmin Moslem

Machine Translation Researcher.

Running TensorBoard with OpenNMT

19 Feb 2021 » nmt

TensorBoard is a tool that provides useful visualization of how training of a deep learning model is going on. It allows you to track and visualize metrics such as accuracy and perplexity. You can use TensorBoard in diverse deep learning frameworks such as TensorFlow and PyTorch. In this tutorial, you will learn how to activate TensorBoard in OpenNMT-tf and OpenNMT-py in different environments.

1- Activating TensorBoard

  • For OpenNMT-tf, TensorBoard is enabled by default. For OpenNMT-py, you need to enable TensorBoard, and optionally customize the log directory. Add these lines to the training configuration YAML file.
    tensorboard: true
    tensorboard_log_dir: logs
    
  • Start your OpenNMT training as usual.

  • Create a screen for TensorBoard: screen -S tensorboard. Note: if you use Google Colab, you do not need screen.

  • Open the directory of the log files. In OpenNMT-tf, by default the log files are in the same folder as the model. In OpenNMT-py, the logs are in a directory with today’s date inside “runs/onmt” or the path you specified for tensorboard_log_dir

  • Start TensorBoard and specify the log directory: tensorboard --logdir="."

  • At this point, you should see a message that TensorBoard is running on localhost http://localhost:6006/ and that’s how to access it from a local browser if you are working on the same machine.

  • Get out of this screen by pressing: Ctrl+A+D.

2- Accessing TensorBoard

Google Colab

You can start TensorBoard within the notebook using magics:

%tensorboard --logdir logs

Exposing TensorBoard to network

You can add the flag --bind_all to your command to be able to open TensorBoard in a local browser with the server IP.

tensorboard --logdir logs --bind_all

ngrok

  • Sign up to ngrok and download the suitable version; for example the one for Linux.

  • Unzip the downloaded ngrok archive.

  • Find your authentication key here and run the command: ./ngrok authtoken <your_authentication_key>

  • Start a new screen: screen -S ngrok

  • Start ngrok on TensorBoard’s default port 6006: ./ngrok https 6006

  • If everything works well, you should see a black screen with “Session Status Online” and other details, including “Forwarding”.

  • Copy the “Forwarding” HTTP or HTTPs and run it in your browser. You should be able to see something like this:

tensorboard

Disclaimer: The ngrok method should be only used for research or demonstration purposes. For corporate and security-sensitive purposes, consult with your team first. Depending on the infrastructure you are using, there might be better methods.

Google Cloud Platform (GCP)

If you are training your models on Google Cloud Platform (GCP), you can rather run TensorBoard locally using the approach explained here and here for example.

You can learn more here about TensorBoard and how to use it in other scenarios.