Notes

Research and Development Workflow on Bracewell

Connecting to Bracewell

CSIRO VPN

Request VPN access. Easily done on-site but can also be done off-site (https://security.csiro.au/offsite.php)

You will most likely be redirected to https://vpn.csiro.au/. When you log in, it will attempt to install the "Cisco AnyConnect Secure Mobility Client", and fail. While you can install this manually, I find the best thing is to install the open-source alternative, openconnect:

$ pacin openconnect networkmanager-openconnect

You can now initiate VPN connections in the command line console with openconnect:

$ sudo openconnect vpn.csiro.au

The Network Manager package makes it really nice and easy to work with

$ pacin networkmanager-openconnect

(you may need to logout and log back in)

SSH

$ ssh -X ident@bracewell.hpc.csiro.au

You can also use the -Y flag instead.

Might be a good idea to set up password-less SSH with public key cryptography.

Interactive node

Use an interactive node for persistence, moderate computing and a well integrated experience. The interactive node for Bracewell is:

bracewell-i1.hpc.csiro.au

Module environment

$ module avail
$ module list
Currently Loaded Modulefiles:
  1) SC                    3) cuda-driver/current   5) intel-fc/16.0.4.258
  2) slurm/16.05.8         4) intel-cc/16.0.4.258
$ module show tensorflow/1.3.0-py35-gpu
-------------------------------------------------------------------
/var/apps_bracewell/modules/modulefiles/tensorflow/1.3.0-py35-gpu:

prepend-path     PATH /apps/tensorflow/1.3.0-py35-gpu/bin
prepend-path     CPATH /apps/tensorflow/1.3.0-py35-gpu/include
prepend-path     PKG_CONFIG_PATH /apps/tensorflow/1.3.0-py35-gpu/lib/pkgconfig
prepend-path     LD_RUN_PATH /apps/tensorflow/1.3.0-py35-gpu/lib
prepend-path     PYTHONPATH /apps/tensorflow/1.3.0-py35-gpu/lib/python3.5/site-packages
setenv       TENSORFLOW_HOME /apps/tensorflow/1.3.0-py35-gpu

load-module  python/3.6.1
load-module  cuda/8.0.61
load-module  cudnn/v6
-------------------------------------------------------------------
$ module load tensorflow/1.3.0-py35-gpu
$ module list
Currently Loaded Modulefiles:
  1) SC                          5) intel-fc/16.0.4.258         9) cudnn/v6
  2) slurm/16.05.8               6) intel-mkl/2017.2.174       10) tensorflow/1.3.0-py35-gpu
  3) cuda-driver/current         7) python/3.6.1
  4) intel-cc/16.0.4.258         8) cuda/8.0.61

Customizing your shell

$ cd ~
$ git clone https://github.com/ltiao/dotfiles.git
$ source bootstrap.sh
$ rm ~/.hushlogin # I don't want to disable the Bracewell ASCII art login screen

While you're here, you might like to set-up your virtualenvwrapper. Add something like the following to your bash_profile:

module load python/3.6.1
export WORKON_HOME=$HOME/.virtualenvs
source $(which virtualenvwrapper_lazy.sh)

Now you can create Python virtualenv s as usual with

$ mkvirtualenv --system-site-packages <virtual-env-name>

Note

Some work still needs to be done to determine how system-wide installed packages are affected when they are upgraded in a virtualenv. For example, I noticed Keras==2.0.3 was install in the system site packages. When I execute pip install Keras==2.0.8 does the virtualenv installed version then take precedence and override the system-wide version? I assume this is the case.

Some dependencies for zsh are missing so you're stuck with Bash.

Note

In batch jobs that use bash (i.e. sinteractive) or if the script you run with sbatch begins with #!/bin/bash, your ~/.bash_profile will be invoked.

Slurm batch system

Time limits:

A time limit of zero requests that no time limit be imposed. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

Batch job scripts

$ sinteractive -h
Usage: sinteractive [-n] [-t] [-p] [-J] [-w] [-g]
Optional arguments:

    -n: Number of tasks to request (default: 1)
        Consider the number of processes you need to run.

    -c: Number of CPUs per task to request (default: 1)
        Consider the number of threads each process requires. A combination of
        number of tasks and CPUs per task are typically required for hybrid
        codes (multi-processes and multi-threads).
        Note that MATLAB's parallel processing requires this to be set. e.g.
        -n 1 -c 4 will allow MATLAB's "local" cluster profile to use 4 workers.

    -t: Wall time to request (default: 2:00:00)

    -m: Memory to request (no default)

    -p: Partition to run job in (no default)

    -J: Job name (default: interactive)

    -w: Node name (no default)

    -g: Request a generic resource e.g. gpu:2

NB: The command that is actually run is printed first so you can copy it
    and run fancier versions with more salloc and srun options if necessary.

e.g.
  sinteractive -n 2 -t 1:00:00 -m 2gb
tia00c at bracewell-i1 in ~
$ sinteractive -n 1 -c 1 -m 50mb -g gpu:1 -t 00:00:30
running: salloc --ntasks-per-node 1 --cpus-per-task=1 --mem 50mb -J interactive -t 00:00:30 --gres gpu:1 srun --pty /bin/bash -l
salloc: Granted job allocation 8186661
srun: Job step created

tia00c at b043 in ~
$

Working with Samples of Distributions over Convolutional Kernels

In [1]:
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
In [2]:
import numpy as np
import tensorflow as tf

import matplotlib.pyplot as plt

from tensorflow.examples.tutorials.mnist import input_data as mnist_data
In [3]:
tf.__version__
Out[3]:
'1.2.1'
In [4]:
sess = tf.InteractiveSession()
In [5]:
mnist = mnist_data.read_data_sets("/home/tiao/Desktop/MNIST")
Extracting /home/tiao/Desktop/MNIST/train-images-idx3-ubyte.gz
Extracting /home/tiao/Desktop/MNIST/train-labels-idx1-ubyte.gz
Extracting /home/tiao/Desktop/MNIST/t10k-images-idx3-ubyte.gz
Extracting /home/tiao/Desktop/MNIST/t10k-labels-idx1-ubyte.gz
In [6]:
# 50 single-channel (grayscale) 28x28 images
x = mnist.train.images[:50].reshape(-1, 28, 28, 1)
x.shape
Out[6]:
(50, 28, 28, 1)
In [7]:
fig, ax = plt.subplots(figsize=(5, 5))

# showing an arbitrarily chosen image
ax.imshow(np.squeeze(x[5], axis=-1), cmap='gray')

plt.show()

Standard 2D Convolution with conv2d

In [8]:
# 32 kernels of size 5x5x1
kernel = tf.truncated_normal([5, 5, 1, 32], stddev=0.1)
kernel.get_shape().as_list()
Out[8]:
[5, 5, 1, 32]
In [9]:
x_conved = tf.nn.conv2d(x, kernel, 
                        strides=[1, 1, 1, 1], 
                        padding='SAME')
x_conved.get_shape().as_list()
Out[9]:
[50, 28, 28, 32]
In [10]:
x_conved[5, ..., 0].eval().shape
Out[10]:
(28, 28)
In [11]:
fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(9, 4))

# showing what the 0th filter looks like
ax1.imshow(kernel[..., 0, 0].eval(), cmap='gray')

# show the previous arbitrarily chosen image
# convolved with the 0th filter
ax2.imshow(x_conved[5, ..., 0].eval(), cmap='gray')

plt.show()

Sample from a Distribution over Kernels

In [12]:
# 8x32 kernels of size 5x5x1
kernels = tf.truncated_normal([8, 5, 5, 1, 32], stddev=0.1)
kernels.get_shape().as_list()
Out[12]:
[8, 5, 5, 1, 32]

Approach 1: Map over samples with conv2d

In [13]:
x_tiled = tf.tile(tf.expand_dims(x, 0), [8, 1, 1, 1, 1])
x_tiled.get_shape().as_list()
Out[13]:
[8, 50, 28, 28, 1]
In [19]:
tf.nn.conv2d(x_tiled[0], kernels[0], 
             strides=[1, 1, 1, 1], 
             padding='SAME').get_shape().as_list()
Out[19]:
[50, 28, 28, 32]
In [15]:
x_conved1 = tf.map_fn(lambda args: tf.nn.conv2d(*args, strides=[1, 1, 1, 1], padding='SAME'),
                      elems=(x_tiled, kernels), dtype=tf.float32)
x_conved1.get_shape().as_list()
Out[15]:
[8, 50, 28, 28, 32]

Approach 2: Flattening

In [16]:
kernels_flat = tf.reshape(tf.transpose(kernels, 
                                       perm=(1, 2, 3, 4, 0)), 
                          shape=(5, 5, 1, 32*8))
kernels_flat.get_shape().as_list()
Out[16]:
[5, 5, 1, 256]
In [17]:
x_conved2 = tf.transpose(tf.reshape(tf.nn.conv2d(x, kernels_flat, 
                                                 strides=[1, 1, 1, 1], 
                                                 padding='SAME'), 
                                    shape=(50, 28, 28, 32, 8)), 
                         perm=(4, 0, 1, 2, 3))
x_conved2.get_shape().as_list()
Out[17]:
[8, 50, 28, 28, 32]
In [18]:
tf.reduce_all(tf.equal(x_conved1, x_conved2)).eval()
Out[18]:
True

Variational Inference with Implicit Approximate Inference Models - @fhuszar's Explaining Away Example Pt. 1 (WIP)

In [1]:
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
In [70]:
import numpy as np
import keras.backend as K

import matplotlib.pyplot as plt
import seaborn as sns

from scipy.stats import logistic, multivariate_normal, norm
from scipy.special import expit

from keras.models import Model, Sequential
from keras.layers import Activation, Add, Dense, Dot, Input
from keras.optimizers import Adam
from keras.utils.vis_utils import model_to_dot

from mpl_toolkits.mplot3d import Axes3D
from matplotlib.animation import FuncAnimation

from IPython.display import HTML, SVG, display_html
from tqdm import tnrange, tqdm_notebook
In [3]:
# display animation inline
plt.rc('animation', html='html5')
plt.style.use('seaborn-notebook')
sns.set_context('notebook')
In [4]:
np.set_printoptions(precision=2,
                    edgeitems=3,
                    linewidth=80,
                    suppress=True)
In [5]:
K.tf.__version__
Out[5]:
'1.2.1'
In [6]:
LATENT_DIM = 2
NOISE_DIM = 3
BATCH_SIZE = 200
PRIOR_VARIANCE = 2.
LEARNING_RATE = 3e-3
PRETRAIN_EPOCHS = 60

Bayesian Logistic Regression (Synthetic Data)

In [7]:
z_min, z_max = -5, 5
In [8]:
z1, z2 = np.mgrid[z_min:z_max:300j, z_min:z_max:300j]
In [9]:
z_grid = np.dstack((z1, z2))
z_grid.shape
Out[9]:
(300, 300, 2)
In [10]:
prior = multivariate_normal(mean=np.zeros(LATENT_DIM), 
                            cov=PRIOR_VARIANCE)
In [11]:
log_prior = prior.logpdf(z_grid)
log_prior.shape
Out[11]:
(300, 300)
In [13]:
np.allclose(log_prior, 
            -.5*np.sum(z_grid**2, axis=2)/PRIOR_VARIANCE \
            -np.log(2*np.pi*PRIOR_VARIANCE))
Out[13]:
True
In [15]:
fig, ax = plt.subplots(figsize=(5, 5))

ax.contourf(z1, z2, log_prior, cmap='magma')

ax.set_xlabel('$w_1$')
ax.set_ylabel('$w_2$')

ax.set_xlim(z_min, z_max)
ax.set_ylim(z_min, z_max)

plt.show()
In [16]:
x = np.array([0, 5, 8, 12, 50])
In [37]:
def log_likelihood(z, x, beta_0=3., beta_1=1.):
    beta = beta_0 + np.sum(beta_1*np.maximum(0, z**3), axis=-1)
    return -np.log(beta) - x/beta
In [44]:
llhs = log_likelihood(z_grid, x.reshape(-1, 1, 1))
llhs.shape
Out[44]:
(5, 300, 300)
In [59]:
fig, axes = plt.subplots(ncols=len(x), nrows=1, figsize=(20, 4))
fig.tight_layout()

for i, ax in enumerate(axes):
    
    ax.contourf(z1, z2, llhs[i,::,::], cmap=plt.cm.magma)

    ax.set_xlim(z_min, z_max)
    ax.set_ylim(z_min, z_max)
    
    ax.set_title('$p(x = {{{0}}} \mid z)$'.format(x[i]))
    ax.set_xlabel('$z_1$')    
    
    if not i:
        ax.set_ylabel('$z_2$')

plt.show()