”;
Many times, we have a need of analysing the text only for the unique words present in the file. So, we need to eliminate the duplicate words from the text. This is achieved by using the word tokenization and set functions available in nltk.
Without preserving the order
In the below example we first tokenize the sentence into words. Then we apply set() function which creates an unordered collection of unique elements. The result has unique words which are not ordered.
import nltk word_data = "The Sky is blue also the ocean is blue also Rainbow has a blue colour." # First Word tokenization nltk_tokens = nltk.word_tokenize(word_data) # Applying Set no_order = list(set(nltk_tokens)) print no_order
When we run the above program, we get the following output −
[''blue'', ''Rainbow'', ''is'', ''Sky'', ''colour'', ''ocean'', ''also'', ''a'', ''.'', ''The'', ''has'', ''the'']
Preserving the Order
To get the words after removing the duplicates but still preserving the order of the words in the sentence, we read the words and add it to list by appending it.
import nltk word_data = "The Sky is blue also the ocean is blue also Rainbow has a blue colour." # First Word tokenization nltk_tokens = nltk.word_tokenize(word_data) ordered_tokens = set() result = [] for word in nltk_tokens: if word not in ordered_tokens: ordered_tokens.add(word) result.append(word) print result
When we run the above program, we get the following output −
[''The'', ''Sky'', ''is'', ''blue'', ''also'', ''the'', ''ocean'', ''Rainbow'', ''has'', ''a'', ''colour'', ''.'']
”;