R session and assign it to an object called comments. Get an overview of the contained variables. What do the variables describe? Why do we have missing data in some of them?
To load the data, you can use the readRDS() function, to get an overview of the contained variables, you can simply use colnames(). To find out more about what the variables mean, you can have a look at the YouTube data API documentation and search for the variable descriptions.
# Load data
comments <- readRDS("../data/RawLWTComments.rds")
# overview of columns
colnames(comments)## [1] "videoId" "textDisplay" "textOriginal"
## [4] "authorDisplayName" "authorProfileImageUrl" "authorChannelUrl"
## [7] "authorChannelId.value" "canRate" "viewerRating"
## [10] "likeCount" "publishedAt" "updatedAt"
## [13] "id" "parentId" "moderationStatus"authorProfileImageUrl, authorChannelUrl, authorChannelUrl.value,video_id,canRate and viewerRating and moderationStatus. Create a new dataframe called Selection containing only the remaining variables.
You can use the subset() function from base R to keep or remove a selection of variables from a dataframe. For more information on how to use it, have a look at its documentation by running ?subset().
# Select only the columns we need
Selection <- subset(comments,select = -c(authorProfileImageUrl,
authorChannelUrl,
authorChannelId.value,
videoId,
canRate,
viewerRating,
moderationStatus))
# Alternatively, you could, of course also use dplyr::select()
# Check selection
colnames(Selection)Check the class of the variable publishedAt in your new dataframe. Is this class suitable for further analysis? If not, change the class to the appropriate one and compute the time difference in publishing dates between the comment in the first row and the comment in the last row.
Do the same transformation for the variable updatedAt
To check the class of the publishedAt variable, you can use the class() function. To check the formatting of the comment timestamp, you can check the YouTube API documentation. To transform character strings into datetime objects in R, you can use the base R function as.POSIXct() or the more convenient anytime() function from the package with the same name.
# check variable class
class(Selection$publishedAt)
# transform to datetime object with as.POISXct
DateTime <- as.POSIXct(Selection$publishedAt,format = "%Y-%m-%dT%H:%M:%SZ")
# transform to datetime object with anytime
library(anytime)
Selection$publishedAt <- anytime(Selection$publishedAt,asUTC = TRUE)
class(Selection$publishedAt)
# compute time difference in publishing time between first and last comment
Selection$publishedAt[1] - Selection$publishedAt[dim(Selection)[1]]
# transform the updatedAt variable as well
Selection$updatedAt <- anytime(Selection$updatedAt,asUTC = TRUE)Check the likeCount variable in your data. Is it suitable for numeric analysis? If not, transform it to the appropriate class and test whether your transformation worked.
You can use the class() function to check the class of an object in R. To change a class, for example from character to numeric, you can use the family of “as”-functions, for example as.numeric()
# check variable class
class(Selection$likeCount)## [1] "character"# transform class
Selection$likeCount <- as.numeric(Selection$likeCount)
# techeck class
class(Selection$likeCount)## [1] "numeric"summary(Selection$likeCount)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.00 0.00 0.00 19.95 1.00 5483.00Check the textOriginal column in your Selection dataframe. There are still hyperlinks in the column that we should remove for later text analysis steps. Extract the hyperlinks from the textOriginal column into a new list called Links. In addition, create a new variable called LinksDel that contains the textOriginal but without the hyperlinks.
The qdapRegex package has many pre-built functions for detecting, removing, and replacing specific character strings. You can, for example, use the rm_url() function to extract and replace hyperlinks. As a reminder: You can check the documentation for this function with ?rm_url().
# load package
library(qdapRegex)
# check column
Selection$textOriginal[276:286]## [1] "https://www.youtube.com/watch?v=Bs4oSQWdWWw"
## [2] "America = \"Census is the most difficult and the largest peacetime operation was undertaken by the government\"\n\n\nIndia = \"That`s Cute\""
## [3] "The US Census Bureau can conduct periodic surveys that could conceivably ask the number of toilets in a home."
## [4] "I live in providence county and I didnt even know this happened lmfao"
## [5] "Trump confused a real estate appraiser and a census taker..."
## [6] "These people who don’t like the census remind me of Ron Swanson"
## [7] "It’s even worse now since Trump signed an executive order that would make a “second census” that would specifically ask if a person is a citizen"
## [8] "Watch this entertaining TikTok video based on the census. Be sure to stay and watch the whole video! https://vm.tiktok.com/7r2pWx/"
## [9] "- Census Man -\nStill a more useful superhero than Aquaman."
## [10] "I am honestly a little confused about the US, aren't everybody required to have some form of nation number? And doesn't that update when you file for where you live? Why would a census being needed? Shouldn't the government already have that information?"
## [11] "How many toilets do thy have, how many desks do thy have , what's your roof made of .................WTF"# extract hyperlinks
Links <- rm_url(Selection$textOriginal, extract = TRUE)
Links[276:286]## [[1]]
## [1] "https://www.youtube.com/watch?v=Bs4oSQWdWWw"
##
## [[2]]
## [1] NA
##
## [[3]]
## [1] NA
##
## [[4]]
## [1] NA
##
## [[5]]
## [1] NA
##
## [[6]]
## [1] NA
##
## [[7]]
## [1] NA
##
## [[8]]
## [1] "https://vm.tiktok.com/7r2pWx/"
##
## [[9]]
## [1] NA
##
## [[10]]
## [1] NA
##
## [[11]]
## [1] NA# remove hyperlinks
LinksDel <- rm_url(Selection$textOriginal)
LinksDel[276:286]## [1] ""
## [2] "America = \"Census is the most difficult and the largest peacetime operation was undertaken by the government\" India = \"That`s Cute\""
## [3] "The US Census Bureau can conduct periodic surveys that could conceivably ask the number of toilets in a home."
## [4] "I live in providence county and I didnt even know this happened lmfao"
## [5] "Trump confused a real estate appraiser and a census taker..."
## [6] "These people who don’t like the census remind me of Ron Swanson"
## [7] "It’s even worse now since Trump signed an executive order that would make a “second census” that would specifically ask if a person is a citizen"
## [8] "Watch this entertaining TikTok video based on the census. Be sure to stay and watch the whole video!"
## [9] "- Census Man - Still a more useful superhero than Aquaman."
## [10] "I am honestly a little confused about the US, aren't everybody required to have some form of nation number? And doesn't that update when you file for where you live? Why would a census being needed? Shouldn't the government already have that information?"
## [11] "How many toilets do thy have, how many desks do thy have , what's your roof made of .................WTF"Check the LinksDel variable to see if there are still emojis contained in the column. For our later analysis, we want to do three things:
To achieve this, we first need a dictionary of emojis and their corresponding textual descriptions in a usable format. Load the emo package and have a look at the contained dataframe jis. Assign it to a new object called EmojiList. Afterwards, source the provided CamelCase.R script (in the scripts folder) to transform the textual description from regular case into CamelCase. Finally, create a new variable called TextEmoDel containing the text without the emoji (hint: you can use the ji_replace_all() function from the emo package for that).
We provide you with a function that capitalizes the first character of each word. The function is called simpleCap() and the name of the script is CamelCase.R. You can load it into your workspace using the source() function and specifying its location. You can find the script containing this function in the scripts folder. Keep in mind that this function only capitalizes the first letters of each word, so you still need to get rid of the extra space characters. The gsub() function is a handy tool for this purpose. You can use the ji_replace_all() function from the emo package to replace emojis with an empty string ("").
# load package
library(emo)
# source script
source("../scripts/CamelCase.R")
# reassign dataframe
EmojiList <- jis
# apply the function to all the names
CamelCaseEmojis <- lapply(jis$name, simpleCap)
# delete empty spaces
CollapsedEmojis <- lapply(CamelCaseEmojis,function(x){gsub(" ", "", x, fixed = TRUE)})
# format back from a list to a vector
EmojiList[,4] <- unlist(CollapsedEmojis)
# check the first 10 entries in the list
EmojiList[1:10,c(1,3,4)]## runes emoji name
## 1 1F600 😀 GrinningFace
## 2 1F601 😁 BeamingFaceWithSmilingEyes
## 3 1F602 😂 FaceWithTearsOfJoy
## 4 1F923 🤣 RollingOnTheFloorLaughing
## 5 1F603 😃 GrinningFaceWithBigEyes
## 6 1F604 😄 GrinningFaceWithSmilingEyes
## 7 1F605 😅 GrinningFaceWithSweat
## 8 1F606 😆 GrinningSquintingFace
## 9 1F609 😉 WinkingFace
## 10 1F60A 😊 SmilingFaceWithSmilingEyes# create text column with removed emojis (and hyperlinks)
TextEmoDel <- ji_replace_all(LinksDel,"")
# Check
LinksDel[c(459,469,694)]## [1] "More bullshit 😷🤢" "😂"
## [3] "That fake Native American 😂😂😩😩"TextEmoDel[c(459,469,694)]## [1] "More bullshit " ""
## [3] "That fake Native American "Ultimately, we want to use our EmojiList dataframe to replace the instances of emojis in our text with the textual descriptions. We can do that by looping over all emojis in all texts and replacing them one at a time. There is a problem however: Some emoji strings are made up of multiple “shorter” emoji strings. If we match parts of a “longer” emoji string and replace it with its textual description, the rest will become unreadable. For this reason, we need to make sure that we replace the emoji from longest to shortest string. Sort the EmojiList dataframe by the length of the emoji column from longest to shortest.
You can count the number of characters in a vector of text using the nchar() function. You can reorder dataframes using the order function and you can reverse an order using the rev() function.
# order from longest to shortest
EmojiList <- EmojiList[rev(order(nchar(jis$emoji))),]
# overview of new order
head(EmojiList[,c(1,3,4)],5)## runes emoji name
## 1862 1F469 200D 2764 FE0F 200D 1F48B 200D 1F469 👩❤️💋👩 Kiss:Woman,Woman
## 1860 1F468 200D 2764 FE0F 200D 1F48B 200D 1F468 👨❤️💋👨 Kiss:Man,Man
## 1858 1F469 200D 2764 FE0F 200D 1F48B 200D 1F468 👩❤️💋👨 Kiss:Woman,Man
## 3570 1F3F4 E0067 E0062 E0077 E006C E0073 E007F 🏴 Wales
## 3569 1F3F4 E0067 E0062 E0073 E0063 E0074 E007F 🏴 ScotlandWe now have a working dictionary for replacing emojis with a textual description! Create a new variable called TextEmoRep as a copy of the LinksDel variable. Next, loop through the ordered EmojiList and, for every element in TextEmoRep, replace the contained emoji with “EMOJI_” followed by their textual description. You can use the rm_default() function from the qdapRegex package to replace custom patterns. Be sure to check the documentation so you can set the appropriate options for the function.
NB: There will be warnings in your console even if you are doing everything right.
Loop through the dictionary sorted from longest to shortest emoji. You need to use a for loop to go through all emojis for all comments, one at a time. The paste() function is useful for adding the prefix “EMOJI_” in front of your textual descriptions. Don’t forget to set the arguments fixed = TRUE, clean = TRUE and trim = FALSE in your call to rm_default()
# assign the column to a new variable
TextEmoRep <- LinksDel
# switch off warnings
options(warn=-1)
# loop through all emojis for all comments
for (i in 1:dim(EmojiList)[1]) {
TextEmoRep <- rm_default(TextEmoRep,
pattern = EmojiList[i,3],
replacement = paste0("EMOJI_",
EmojiList[i,4],
" "),
fixed = TRUE,
clean = FALSE,
trim = FALSE)
}
# check results
LinksDel[c(459,469,694)]## [1] "More bullshit 😷🤢" "😂"
## [3] "That fake Native American 😂😂😩😩"TextEmoRep[c(459,469,694)]## [1] "More bullshit EMOJI_FaceWithMedicalMask EMOJI_NauseatedFace "
## [2] "EMOJI_FaceWithTearsOfJoy "
## [3] "That fake Native American EMOJI_FaceWithTearsOfJoy EMOJI_FaceWithTearsOfJoy EMOJI_WearyFace EMOJI_WearyFace "We now have the original text column, and the text column with removed hyperlinks in which emojis are replaced with their textual descriptions (TextEmoRep). We need one more variable that only contains the textual descriptions of the emojis. You can use our predefined function ExtractEmoji() from the scripts folder to create this variable.
Use the source() function to source the ExtractEmoji.R script from the scripts folder and then sapply() the ExtractEmoji() function to the variable TextEmoRep. To remove useless rownames from the extracted Emojis, you can set names(Emoji) to NULL
# source script containing the function
source("../scripts/ExtractEmoji.R")
# apply function & remove rownames
Emoji <- sapply(TextEmoRep,ExtractEmoji)
names(Emoji) <- NULL
# check results
LinksDel[c(459,469,694)]## [1] "More bullshit 😷🤢" "😂"
## [3] "That fake Native American 😂😂😩😩"TextEmoRep[c(459,469,694)]## [1] "More bullshit EMOJI_FaceWithMedicalMask EMOJI_NauseatedFace "
## [2] "EMOJI_FaceWithTearsOfJoy "
## [3] "That fake Native American EMOJI_FaceWithTearsOfJoy EMOJI_FaceWithTearsOfJoy EMOJI_WearyFace EMOJI_WearyFace "Emoji[c(459,469,694)]## [1] "EMOJI_FaceWithMedicalMask EMOJI_NauseatedFace "
## [2] "EMOJI_FaceWithTearsOfJoy "
## [3] "EMOJI_FaceWithTearsOfJoy EMOJI_FaceWithTearsOfJoy EMOJI_WearyFace EMOJI_WearyFace "We now have selected all the variables we need, brought them into the right formats, cleaned the text, and extracted some additional information from it. As a final step, create a new dataframe called df that contains the following variables:
Selection$authorDisplayName
Selection$textOriginal
TextEmoRep
TextEmoDel
Emoji
Selection$likeCount
Links
Selection$publishedAt
Selection$updatedAt
Selection$parentId
Selection$id
Set the following names for the column in the new dataframe:
Author
Text
TextEmojiReplaced
TextEmojiDeleted
Emoji
LikeCount
URL
Published
Updated
ParentId
CommentID
Save the new dataframe as an .rds file with the name “ParsedLWTComments.Rds”
You can use the cbind.data.frame() function to paste together multiple columns into a dataframe. Note: You need to set the argument stringsAsFactors = FALSE if your R version is < 4.0.0 to prevent strings from being interpreted as factors. In addition, the variables Links and Emoji are lists and can contain multiple values per row. For this reason, you need to enclose them with the I() function to be able to put them into a dataframe. You can save your result using the saveRDS() function.
# create df dataframe (use I() function to enclose Emoji and Links)
df <- cbind.data.frame(Selection$authorDisplayName,
Selection$textOriginal,
TextEmoRep,
TextEmoDel,
I(Emoji),
Selection$likeCount,
I(Links),
Selection$publishedAt,
Selection$updatedAt,
Selection$parentId,
Selection$id,
stringsAsFactors = FALSE)
# set column names
names(df) <- c("Author",
"Text",
"TextEmojiReplaced",
"TextEmojiDeleted",
"Emoji",
"LikeCount",
"URL",
"Published",
"Updated",
"ParentId",
"CommentID")
# delete row names
row.names(df) <- NULL
# save dataframe
saveRDS(df, file = "../data/ParsedLWTComments.rds")