mlatoz

Recurrent Neural Networks (RNNs)

Resources

If you want to add more value to this section, we recommend checking out:

• The Ultimate Guide to Recurrent Neural Networks

These resources will compliment this course, so we hope you enjoy!

---

Plan of Attack

• What we will learn in this section:
  • The idea behind Recurrent Neural Networks
  • The Vanishing Gradient Problem
  • Long Short-Term Memory (LSTM)
  • Practical Intuition
    
    
  • EXTRA: LSTM Variations

---

The Idea Behind Recurrent Neural Networks

Supervised	Artificial Neural Networks	Used for Regression & Classification
	Convolutional Neural Networks	Used for Computer Vision
	Recurrent Neural Networks	Used for Time Series Analysis

Unsupervised	Self-Organizing Maps	Used for Feature Detection
	Deep Boltzmann Machines	Used for Recommendation Systems
	AutoEncoders	Used for Recommendation Systems

---

Additional Reading

Sunspring (movie, 2016)

• Directed By Oscar Sharp
• Written By Benjamin

Sunspring (movie, 2016) - Oscar Sharp and Benjamin

---

The Vanishing Gradient Problem

Wrec -> Recurrent Weight

Wrec ~ small -> Vanishing

Wrec ~ large -> Exploding

Solutions:

1. Exploding Gradient
   • Truncated Backpropagation
   • Penalties
   • Gradient Clipping
2. Vanishing Gradient
   • Weight Initialization
   • Echo State Networks
   • Long Short-Term Memory Networks (LSTMs)

---

Additional Reading

Untersuchungen zu dynamischen neuronalen Netzen

By Sepp (Josef) Hochreiter (1991)

Untersuchungen zu dynamischen neuronalen Netzen - Sepp (Josef) Hochreiter

---

Additional Reading

Learning Long-Term Dependencies with Gradient Descent is Difficult

By Yoshua Bengio et al. (1994)

Learning Long-Term Dependencies with Gradient Descent is Difficult - Yoshua Bengio

---

Additional Reading

On The Difficulty of Training Recurrent Neural Networks

By Razvan Pascanu et al. (2013)

On The Difficulty of Training Recurrent Neural Networks - Razvan Pascanu

---

LSTMs

• Today:
  • A bit of History
  • LSTM Architecture
  • Example Walkthrough

Reference:

Understanding LSTM Networks

---

Additional Reading

Long Short-Term Memory

By Sepp Hochreiter & Jurgen Schmidhuber (1997)

Long Short-Term Memory - Sepp Hochreiter & Jurgen Schmidhuber

---

Additional Reading

Understanding LSTM Networks

By Christopher Olah (2015)

Understanding LSTM Networks - Christopher Olah

---

LSTM Practical Intuition

Reference:

The Unreasonable Effectiveness of Recurrent Neural Networks

---

Additional Reading

The Unreasonable Effectiveness of Recurrent Neural Networks

By Andrej Karpathy (2015)

The Unreasonable Effectiveness of Recurrent Neural Networks - Andrej Karpathy

---

Additional Reading

Visualizing and Understanding Recurrent Networks

By Andrej Karpathy et al. (2015)

Visualizing and Understanding Recurrent Networks - Andrej Karpathy

---

LSTM Variations

Additional Reading

LSTM: A Search Space Odyssey

By Klaus Greff et al. (2015)

LSTM: A Search Space Odyssey - Klaus Greff

---

«Previous