2. Machine Learning Algorithms

Currently Veles ML algorithms are Neural Networks as Znicz plugin and Genetic Algorithm as Genetic in Veles core. Znicz is a submodule of Veles.

2.1. Neural Networks

2.1.1. Fully-connected Neural Networks

_images/fully_connected.png

Status:

  1. All necessary units developed for CUDA, OPENCL and NUMPY
  2. Workflow was created and tested
  3. Examples of Workflows:
    1. Softmax NN custom workflow: MnistSimple/mnist.py MnistSimple/mnist_config.py. Download: https://velesnet.ml/forge/fetch?name=MnistSimple&version=master
    2. Softmax NN with standard workflow help: MNIST/mnist.py MNIST/mnist_config.py. Download: https://velesnet.ml/forge/fetch?name=MNIST&version=master
    3. MSE NN with standard workflow help: Kanji/kanji.py Kanji/kanji_config.py Download: https://velesnet.ml/forge/fetch?name=Kanji&version=master
  4. Usage example: MNIST/mnist_forward.py. Download: https://velesnet.ml/forge/fetch?name=MNIST&version=master
  5. Results: 1.48% validation errors on MNIST (Download snapshot: https://s3-eu-west-1.amazonaws.com/veles.forge/MnistSimple/mnist_validation_1.48_train_0.00.4.pickle.gz)

2.1.2. Convolutional Neural Networks

_images/conv.png

Status:

  1. All necessary units developed for CUDA, OPENCL and NUMPY

  2. Workflow was created and tested

  3. Examples of Workflow:

    1. CIFAR/cifar.py CIFAR/cifar_caffe_config.py. Download: https://velesnet.ml/forge/fetch?name=CIFAR10&version=master
    2. AlexNet/imagenet_workflow.py AlexNet/imagenet_workflow_config.py. Download: https://velesnet.ml/forge/fetch?name=AlexNet&version=master
  4. Usage example: AlexNet/ImagenetForward.ipynb. Download: https://velesnet.ml/forge/fetch?name=AlexNet&version=master

  5. Results: 17.21% validation errors on CIFAR10 (Download snapshot: https://s3-eu-west-1.amazonaws.com/veles.forge/CIFAR10/cifar_caffe_validation_17.21_train_8.31.4.pickle.gz )

    35.10% validation errors on STL10 (Download snapshot: https://s3-eu-west-1.amazonaws.com/veles.forge/Stl10/stl10_validation_35.10_train_0.12.4.pickle.gz )

2.1.3. Autoencoder Neural Networks

_images/autoencoder.png

Status:

  1. All necessary units developed for CUDA, OPENCL and NUMPY
  2. Workflow was created and tested
  3. Examples of Workflow:
    1. MnistAE/mnist_ae.py MnistAE/mnist_ae_config.py. Download: https://velesnet.ml/forge/fetch?name=MnistAE&version=master
    2. VideoAE/video_ae.py VideoAE/video_ae_config.py. Download: https://velesnet.ml/forge/fetch?name=VideoAE&version=master
  4. Results: 0.5478 validation rmse on MNIST (Download snapshot: https://s3-eu-west-1.amazonaws.com/veles.forge/MnistAE/mnist_ae_0.5478_0.5482.4.pickle )

2.1.4. Kohonen maps

_images/kohonen.png

Status:

  1. All necessary units developed only for OPENCL and NUMPY. CUDA support is in progress.
  2. Workflow was created and tested
  3. Examples of Workflow:
    1. SpamKohonen/spam_kohonen.py SpamKohonen/spam_kohonen_config.py. Download: https://velesnet.ml/forge/fetch?name=SpamKohonen&version=master
    2. DemoKohonen/kohonen.py DemoKohonen/kohonen_config.py. Download: https://velesnet.ml/forge/fetch?name=DemoKohonen&version=master

2.1.5. Convolutional Autoencoder Neural Networks

_images/conv.png

Status:

  1. All necessary units developed for CUDA, OPENCL and NUMPY

  2. Workflow was created and tested

  3. Example of Workflow (careful: a lot of code and not the best topology):

    ImagenetAE/imagenet_ae.py ImagenetAE/imagenet_ae_config.py. Download: https://velesnet.ml/forge/fetch?name=ImagenetAE&version=master

2.1.6. Restricted Boltzmann Machine

_images/rbm.png

Status:

  1. All necessary units developed for NUMPY
  2. Workflow was created, but not tested.
  3. The creation of working example is in progress.

2.1.7. Reccurent Neural Networks

_images/rnn.png

Status:

  1. All necessary units developed for CUDA, OPENCL and NUMPY
  2. Workflow was created, but not tested.
  3. The creation of working example is in progress.

2.1.8. Long short-term memory

_images/lstm.png

Status:

  1. All necessary units developed for CUDA, OPENCL and NUMPY
  2. Workflow was created, but not tested.
  3. The creation of working example is in progress.

2.2. Genetic algorithm

_images/genetic.png

We use Genetic Algorithm for optimization of Machine Learning hyperparameters (–optimize mode). Example of using Genetic Algorithm for other purposes: veles/samples/GeneticExample (It will be on Forge VelesForge soon)

2.3. Extras

What else we have:

  1. Deconvolution, Depooling
  2. Dropout
  3. Learning rate adjusting
  4. Activation function customization (like SinCos activation function)
  5. Regularization (L1, L2, custom)
  6. Last Models: AlexNet, VGG (AlexNet/imagenet_workflow.py. AlexNet/imagenet_workflow_vgga_config.py Download: https://velesnet.ml/forge/fetch?name=AlexNet&version=master )
  7. Loss functions: mse, softmax
  8. Stochastic gradient descent solver with momentum
  9. AdaGrad/AdaDelta solvers (done, but not tested)
  10. Grouping
  11. Rollback (It saves the best state and returns to it (if some iterations was not successfull) and changes learning rate)
  12. Smart automatic weights/bias initial filling
  13. Ability to change hyperparameters (learning rate, weights decay, momentum, etc.) for each layer
  14. Padding/stride (stride is named “sliding” in Veles)