목록Data Science/Deep Learning (8)
Code&Data Insights
[Deep Learning] Recurrent Neural Networks (RNN)
Recurrent Neural Networks (RNN) : the current output depends on all the previous inputs *SEQUENCE* Xt = current input Ht = previous state W = set of learnable parameters Comparison between a hidden RNN layer and a linear layer - Hidden RNN Layer : the memory that maintains info about past inputs in the sequence => The hidden state is updated at each time stop and can capture short-term dependenc..
[Deep Learning] Regularization - Dropout | Data Augmentation | Multitask Learning
Regularization : techniques used to control the complexity of models and prevent overfitting are known as regularization techniques => Increase generalization ability! => Deep Neural Networks are models with large capacity and thus more prone to overfitting problems!!! Data Augmentation : create fake training data by applying some transformations to the original data - Used for classficiation pr..
[Deep Learning] Convolutional Neural Network (CNN)
What is a Convolution? : a standard operation, long used in compression, signal processing, computer vision, and image processing Convolution = Filtering = Feature Extraction Main difference with the MLP 1) Local Connection : Local connections can capture local patterns better than fully-connected models -> search for all the local patterns by sliding the same kernel -> have the chance to react ..
[Deep Learning] Vanishing and Exploding Gradients | Weight Initialization | Batch Normalization | Layer Normalization
Vanishing Gradients - usually occurs due to the fact that some activation functions squash the input into small values result in small gradients that result in negligible updates to the weights of the model - Or sometimes the input values are small to begin with : When backpropagation the gradient through long chains of computations, the gradient gets smaller and smaller - causes the gradient of..
[Deep Learning] Advanced Optimization Methods - Momentum | Adaptive Learning Rate | AdaGrad | RMSProp | Adam | Distributed Synchronous SGD
Benefits of Advanced optimization methods - Faster Convergence - Improved Stability - Avoiding Local Minima - Better Generalization Momentum : accumulates an exponentially-decaying moving average of the past gradients - NOT ONLY denpends on learning rate, but ALSO past gradients (SDG with Batch) If the previous update vt is very different from the current gradient => little update If previous up..
[Deep Learning] Optimization - Gradient Descent | Stochastic Gradient Descent | Mini-batch SGD
Optimization - Training a machine learning model often requires solving Optimization problem => have to find the parameters of the function f that minimizes the loss function using the training data. Problem in Optimization in Multi dimensional Spaces - TOO MANY CRITICAL POINT! (critical points where f'(x) = 0) => local minima, maxima, and saddle points How to Solve Optimization Problem? Solutio..
[Deep Learning] Neural Networks - Perceptron | MLP | Activation Function | Backpropagation | Gradient Descent
Neural Networks Neural Network : A neural network is composed of neurons that take inputs and calculate outputs through weights and activation functions. Through this process, it learns patterns in data and gains the ability to make predictions. Learning in Neural Network 1) Feed Forward 2) Compute Loss 3) Backpropagate ( = chain rule) : the method to compute the gradient efficiently 4) Gradient..
[Deep Learning] What is Deep Learning?
[ What is Deep Learning? ] - deep learning is a subset of machine learning. ( subset of AI ) - a type of ML inspired by the structure of the human brain => 'Artificial Neural Network' - most efficient way to deal with unstructured data The difference between ML and DL - ML : The features should be given - DL : The features are picked out by the neural network without human intervention => cost h..