Calabar

使Colab更便捷的实用工具。

内容

• 安装
• 使用方法
  • 与Google Drive协作
  • 与归档文件协作
  • 通过Gmail发送通知
  • 其他
• 完整使用示例

安装

!pip install calabar

使用方法

与Google Drive协作

挂载您的驱动器

from calabar import MyDrive

drive = MyDrive('/drive')
drive.mount()

从挂载的驱动器复制文件/文件夹，例如，一些数据，上一次训练中的模型检查点等

file2copy = '/drive/My Drive/data.tar'
dest = '/content'

drive.cp(file2copy, dest)

注意： 在挂载过程中，内部Colab工具将在/drive内创建'My Drive'文件夹，所以别忘了将'My Drive'添加到文件路径中。

从Colab导出文件到驱动器，例如，训练日志、模型检查点等

from calabar import SaveToDrive

export = SaveToDrive()

export.to_drive('file_on_colab')
export.to_drive('another_file')

注意： 简单地复制到挂载的驱动器将没有任何效果。

与归档文件协作

当前支持tar/tar.gz和zip归档文件的解压缩。

from calabar import untar

untar('data.tar', 'to_destination')

from calabar import unzip

unzip('data.zip', 'to_destination')

通过Gmail发送通知

注意： 使用此选项，您需要在您的Gmail账户上允许“不安全的应用”

https://www.google.com/settings/security/lesssecureapps

安全警告： 持续发送电子邮件（例如，在每个epoch后发送电子邮件）需要在内存中存储您的Gmail账户密码，这样您在想要发送电子邮件时不必每次都重新登录到SMTP服务器。但是，Calabar使用安全的TLS连接。无论如何，如果您因为“不安全的应用程序”和SMTP服务器而感到不安全，或者您只是不喜欢这样做，您可能不会使用它或使用另一个“不安全”的账户（开启“不安全的应用程序”选项）并将电子邮件发送到您的主账户，如下所示

from calabar import Email

from_addr = 'johndoe@gmail.com'  # email with "allow less secure", this account's SMTP server will be used as a sender
to_addrs = ['another@gmail.com',]  # a list of receivers

email = Email(from_addr, to_addrs)

msg_subject = 'Subject of the email'
msg_body = 'Email body.'

email.send(msg_subject, msg_body)

使用 | 内容

其他

from calabar.utils import *

# Print system info, gpu, and installed pytorch version

print_sysinfo()

# Get current GPU RAM usage:

print(get_gpu_usage())

# Get disk usage:

print(get_disk_usage())

# Get current time as formatted string in %Y-%m-%d-%H-%M:

print(current_time())

# Get distro description:

print(get_distro_descr())

其他 | 使用 | 内容

完整使用示例

以下示例展示了如何将Calabar集成到您的训练流程中。

from calabar import MyDrive, SaveToDrive, untar, unzip, Email
from calabar.utils import *


# Authorize to Drive to allow mounting
drive = MyDrive('/drive')
drive.mount()

# Authorize to Drive to to allow exporting | NOTE: you can use here another Drive
export = SaveToDrive()

from_addr = 'johndoe@gmail.com'
to_addrs = ['another@gmail.com']

email = Email(from_addr, to_addrs)

# Prepare your data 

# Copy your data:
drive.cp('/drive/My Drive/data.tar', '/content')

# Untar/unzip it:
untar('/content/data.tar', '/content')

# ...
# Model definition, training, validation functions etc...
# ...

# Main loop:

# Notification about beginning of the model training process 

start_sub = 'Starting training...'
start_msg = f'Training (your model descr) has started at {current_time()}\
            \nCurrent GPU usage: {get_gpu_usage()}\
            \nCurrent disk usage: {get_disk_usage()}'

email.send(start_sub, start_msg)

for epoch in n_epochs:
    train_loss, train_acc = train(...)
    val_loss, val_ acc = validate(...)
      
    # Here you're saving your model checkpoint
    checkpoint_name = f'{epoch}-{val_loss:.3f}-{acc:.3f}-mymodel.checkpoint'  # this is just example
    save_checkpoint(f'/content/checkpoints/{checkpoint_name}')
    
    # Here you're loading it to your Drive:
    export.to_drive(f'/content/checkpoints/{checkpoint_name}')
    
    # Epoch is over, model checkpoint is exported to your drive, let's notify you about current success:
    
    epoch_sub = f'{epoch}/{n_epoch} has finished!'
    epoch_msg = f'{epoch}/{n_epoch} has finished successfully at {current_time()}\
                \ntrain_loss: {train_loss:.3f}|val_loss: {val_loss:.3f}|train_acc: {train_acc:.3f}|val_acc: {val_acc:.3f}\
                \nModel checkpoint {checkpoint_name} successfully loaded to Drive\
                \nCurrent GPU usage: {get_gpu_usage()}\
                \nCurrent disk usage: {get_disk_usage()}'

    email.send(epoch_sub, epoch_msg)
    
# Notification email about model training completion.

end_sub = 'Training has finished!'
end_msg = f'Training (your model descr) has finished at {current_time()}\
            \nCurrent GPU usage: {get_gpu_usage()}\
            \nCurrent disk usage: {get_disk_usage()}'

email.send(end_sub, end_msg)

完整使用示例 | 其他 | 使用 | 内容

待办事项

• tar/zip归档
• 更多文档
• 更多示例