Word embeddings | Text | TensorFlow
Skip to main content English Español – América Latina Français Indonesia Italiano Polski Português – Brasil Tiếng Việt Türkçe Русский עברית العربيّة فارسی हिंदी বাংলা ภาษาไทย 中文 – 简体 日本語 한국어 GitHub Sign in Text TensorFlow Resources Text Tutorials Word embeddings Stay organized with collections Save and categorize content based on your preferences.
View on TensorFlow.org Run in Google Colab View source on GitHub Download notebook This tutorial contains an introduction to word embeddings. You will train your own word embeddings using a simple Keras model for a sentiment classification task, and then visualize them in the Embedding Projector (shown in the image below). Representing text as numbers Machine learning models take vectors (arrays of numbers) as input. When working with text, the first thing you must do is come up with a strategy to convert strings to numbers (or to "vectorize" the text) before feeding it to the model.
In this section, you will look at three strategies for doing so. One-hot encodings As a first idea, you might "one-hot" encode each word in your vocabulary. Consider the sentence "The cat sat on the mat". The vocabulary (or unique words) in this sentence is (cat, mat, on, sat, the). To represent each word, you will create a zero vector with length equal to the vocabulary, then place a one in the index that corresponds to the word.
This approach is shown in the following diagram. To create a vector that contains the encoding of the sentence, you could then concatenate the one-hot vectors for each word. Encode each word with a unique number A second approach you might try is to encode each word using a unique number. Continuing the example above, you could assign 1 to "cat", 2 to "mat", and so on. You could then encode the sentence "The cat sat on the mat" as a dense vector like [5, 1, 4, 3, 5, 2].
This approach is efficient. Instead of a sparse vector, you now have a dense one (where all elements are full). There are two downsides to this approach, however: The integer-encoding is arbitrary (it does not capture any relationship between words). An integer-encoding can be challenging for a model to interpret. A linear classifier, for example, learns a single weight for each feature. Because there is no relationship between the similarity of any two words and the similarity of their encodings, this feature-weight combination is not meaningful.
Word embeddings Word embeddings give us a way to use an efficient, dense representation in which similar words have a similar encoding. Importantly, you do not have to specify this encoding by hand. An embedding is a dense vector of floating point values (the length of the vector is a parameter you specify). Instead of specifying the values for the embedding manually, they are trainable parameters (weights learned by the model during training, in the same way a model learns weights for a dense layer).
It is common to see word embeddings that are 8-dimensional (for small datasets), up to 1024-dimensions when working with large datasets. A higher dimensional embedding can capture fine-grained relationships between words, but takes more data to learn. Above is a diagram for a word embedding. Each word is represented as a 4-dimensional vector of floating point values. Another way to think of an embedding is as "lookup table".
After these weights have been learned, you can encode each word by looking up the dense vector it corresponds to in the table. Setup import io import os import re import shutil import string import tensorflow as tf from tensorflow.keras import Sequential from tensorflow.keras.layers import Dense , Embedding , GlobalAveragePooling1D from tensorflow.keras.layers import TextVectorization 2022-12-14 12:16:56.601690: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 12:16:56.601797: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 12:16:56.601808: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries.
If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. Download the IMDb Dataset You will use the Large Movie Review Dataset through the tutorial. You will train a sentiment classifier model on this dataset and in the process learn embeddings from scratch. To read more about loading a dataset from scratch, see the Loading text tutorial . Download the dataset using Keras file utility and take a look at the directories. url = "https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz" dataset = tf . keras . utils . get_file ( "aclImdb_v1.tar.gz" , url , untar = True , cache_dir = '.' , cache_subdir = '' ) dataset_dir = os . path . join ( os . path . dirname ( dataset ), 'aclImdb' ) os . listdir ( dataset_dir ) Downloading data from https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz 84125825/84125825 [==============================] - 3s 0us/step ['test', 'imdb.vocab', 'README', 'train', 'imdbEr.txt'] Take a look at the train/ directory.
It has pos and neg folders with movie reviews labelled as positive and negative respectively. You will use reviews from pos and neg folders to train a binary classification model. train_dir = os . path . join ( dataset_dir , 'train' ) os . listdir ( train_dir ) ['pos', 'neg', 'unsupBow.feat', 'urls_pos.txt', 'unsup', 'urls_neg.txt', 'urls_unsup.txt', 'labeledBow.feat'] The train directory also has additional folders which should be removed before creating training dataset. remove_dir = os . path . join ( train_dir , 'unsup' ) shutil . rmtree ( remove_dir ) Next, create a tf.data.Dataset using tf.keras.utils.text_dataset_from_directory .
You can read more about using this utility in this text classification tutorial . Use the train directory to create both train and validation datasets with a split of 20% for validation. batch_size = 1024 seed = 123 train_ds = tf . keras . utils . text_dataset_from_directory ( 'aclImdb/train' , batch_size = batch_size , validation_split = 0.2 , subset = 'training' , seed = seed ) val_ds = tf . keras . utils . text_dataset_from_directory ( 'aclImdb/train' , batch_size = batch_size , validation_split = 0.2 , subset = 'validation' , seed = seed ) Found 25000 files belonging to 2 classes.
Using 20000 files for training. Found 25000 files belonging to 2 classes. Using 5000 files for validation. Take a look at a few movie reviews and their labels (1: positive, 0: negative) from the train dataset. for text_batch , label_batch in train_ds . take ( 1 ): for i in range ( 5 ): print ( label_batch [ i ] . numpy (), text_batch . numpy ()[ i ]) 0 b"Oh My God! Please, for the love of all that is holy, Do Not Watch This Movie!
It it 82 minutes of my life I will never get back. Sure, I could have stopped watching half way through. But I thought it might get better. It Didn't. Anyone who actually enjoyed this movie is one seriously sick and twisted individual. No wonder us Australians/New Zealanders have a terrible reputation when it comes to making movies. Everything about this movie is horrible, from the acting to the editing. I don't even normally write reviews on here, but in this case I'll make an exception.
I only wish someone had of warned me before I hired this catastrophe" 1 b'This movie is SOOOO funny!!! The acting is WONDERFUL, the Ramones are sexy, the jokes are subtle, and the plot is just what every high schooler dreams of doing to his/her school. I absolutely loved the soundtrack as well as the carefully placed cynicism. If yo
The rest of this page is on the live site.