Experiment tracking with WandB¶
Weights and Biases (WandB) is a service for logging model performance and storing model outputs in a cloud environment that can be accessed via a neat web UI.
Initializing wandb¶
You'll need to login to wandb to use it's features. You can do this manually on each instance, or you can use an environment variable. You'll need your API key in either case, which you can get from wandb.ai/settings.
Command line login¶
wandb login
Then you'll copy/paste your API key. If you don't want to remember this, we've added an entry in the Makefile you can call with make wandb.
With environment variables¶
Specify the following environment variable in compute environments where it's not easy to directly log in (say, in automated model training runs).
WANDB_API_KEY=${YOUR_API_KEY}
Configuring wandb¶
The majority of wandb logic is handled internally in train.py. We configure all of our logging options via our model configuration yml files. The default configurations include:
enabled: true
team: forest-observatory
project: null
model_id: null
notes: null
sync_tensorboard: true
enabled: set this to false to disable logging via wandb (useful for local debugging or automated testing)
team: the namespace where the model runs will be logged. Set it to your wandb username if you're testing models that you don't need logged and stored.
project: we track models under separate projects, and project IDs should be set for any operational model. If null, it will be tracked in the uncategorized project.
model_id: a unique identifier to lookup the outputs from this model training session. If null, an id is automatically generated.
notes: a message about this model run. Nothing is logged if null.
sync_tensorboard: synchronize reporting between wandb and tensorboard.
Additional configurations logged by myco¶
In addition to the above user-configurable parameters, myco will automatically log a series of metrics during model training. These relate to the compute environment (number of gpus, whether you're running on Compute Engine) and to the model parameters (architecture type, input datasets).