Exercise 1

Load the comment data you scraped in the previous set of exercises into your R session and assign it to an object called comments. Get an overview of the contained variables. What do the variables describe? Why do we have missing data in some of them?

To load the data, you can use the readRDS() function, to get an overview of the contained variables, you can simply use colnames(). To find out more about what the variables mean, you can have a look at the YouTube data API documentation and search for the variable descriptions.

Exercise 2

We want to remove the variables authorProfileImageUrl, authorChannelUrl, authorChannelUrl.value,video_id,canRate and viewerRating and moderationStatus. Create a new dataframe called Selection containing only the remaining variables.

You can use the subset() function from base R to keep or remove a selection of variables from a dataframe. For more information on how to use it, have a look at its documentation by running ?subset().

Exercise 3

Check the class of the variable publishedAt in your new dataframe. Is this class suitable for further analysis? If not, change the class to the appropriate one and compute the time difference in publishing dates between the comment in the first row and the comment in the last row.

Do the same transformation for the variable updatedAt

To check the class of the publishedAt variable, you can use the class() function. To check the formatting of the comment timestamp, you can check the YouTube API documentation. To transform character strings into datetime objects in R, you can use the base R function as.POSIXct() or the more convenient anytime() function from the package with the same name.

Exercise 4

Check the likeCount variable in your data. Is it suitable for numeric analysis? If not, transform it to the appropriate class and test whether your transformation worked.

You can use the class() function to check the class of an object in R. To change a class, for example from character to numeric, you can use the family of “as”-functions, for example as.numeric()

Exercise 5

Check the textOriginal column in your Selection dataframe. There are still hyperlinks in the column that we should remove for later text analysis steps. Extract the hyperlinks from the textOriginal column into a new list called Links. In addition, create a new variable called LinksDel that contains the textOriginal but without the hyperlinks.

The qdapRegex package has many pre-built functions for detecting, removing, and replacing specific character strings. You can, for example, use the rm_url() function to extract and replace hyperlinks. As a reminder: You can check the documentation for this function with ?rm_url().

Exercise 6

Check the LinksDel variable to see if there are still emojis contained in the column. For our later analysis, we want to do three things:

  1. Create one column without hyperlinks and emojis for easier text mining
  2. Create one column where emojis are replaced by a textual description for easier text mining
  3. Create one column containing only the textual description of emojis

To achieve this, we first need a dictionary of emojis and their corresponding textual descriptions in a usable format. Load the emo package and have a look at the contained dataframe jis. Assign it to a new object called EmojiList. Afterwards, source the provided CamelCase.R script (in the scripts folder) to transform the textual description from regular case into CamelCase. Finally, create a new variable called TextEmoDel containing the text without the emoji (hint: you can use the ji_replace_all() function from the emo package for that).

We provide you with a function that capitalizes the first character of each word. The function is called simpleCap() and the name of the script is CamelCase.R. You can load it into your workspace using the source() function and specifying its location. You can find the script containing this function in the scripts folder. Keep in mind that this function only capitalizes the first letters of each word, so you still need to get rid of the extra space characters. The gsub() function is a handy tool for this purpose. You can use the ji_replace_all() function from the emo package to replace emojis with an empty string ("").

Exercise 7

Ultimately, we want to use our EmojiList dataframe to replace the instances of emojis in our text with the textual descriptions. We can do that by looping over all emojis in all texts and replacing them one at a time. There is a problem however: Some emoji strings are made up of multiple “shorter” emoji strings. If we match parts of a “longer” emoji string and replace it with its textual description, the rest will become unreadable. For this reason, we need to make sure that we replace the emoji from longest to shortest string. Sort the EmojiList dataframe by the length of the emoji column from longest to shortest.

You can count the number of characters in a vector of text using the nchar() function. You can reorder dataframes using the order function and you can reverse an order using the rev() function.

Exercise 8

We now have a working dictionary for replacing emojis with a textual description! Create a new variable called TextEmoRep as a copy of the LinksDel variable. Next, loop through the ordered EmojiList and, for every element in TextEmoRep, replace the contained emoji with “EMOJI_” followed by their textual description. You can use the rm_default() function from the qdapRegex package to replace custom patterns. Be sure to check the documentation so you can set the appropriate options for the function.

NB: There will be warnings in your console even if you are doing everything right.

Loop through the dictionary sorted from longest to shortest emoji. You need to use a for loop to go through all emojis for all comments, one at a time. The paste() function is useful for adding the prefix “EMOJI_” in front of your textual descriptions. Don’t forget to set the arguments fixed = TRUE, clean = TRUE and trim = FALSE in your call to rm_default()

Exercise 9

We now have the original text column, and the text column with removed hyperlinks in which emojis are replaced with their textual descriptions (TextEmoRep). We need one more variable that only contains the textual descriptions of the emojis. You can use our predefined function ExtractEmoji() from the scripts folder to create this variable.

Use the source() function to source the ExtractEmoji.R script from the scripts folder and then sapply() the ExtractEmoji() function to the variable TextEmoRep. To remove useless rownames from the extracted Emojis, you can set names(Emoji) to NULL

Exercise 10

We now have selected all the variables we need, brought them into the right formats, cleaned the text, and extracted some additional information from it. As a final step, create a new dataframe called df that contains the following variables:

  • Selection$authorDisplayName

  • Selection$textOriginal

  • TextEmoRep

  • TextEmoDel

  • Emoji

  • Selection$likeCount

  • Links

  • Selection$publishedAt

  • Selection$updatedAt

  • Selection$parentId

  • Selection$id

Set the following names for the column in the new dataframe:

  • Author

  • Text

  • TextEmojiReplaced

  • TextEmojiDeleted

  • Emoji

  • LikeCount

  • URL

  • Published

  • Updated

  • ParentId

  • CommentID

Save the new dataframe as an .rds file with the name “ParsedLWTComments.Rds”

You can use the cbind.data.frame() function to paste together multiple columns into a dataframe. Note: You need to set the argument stringsAsFactors = FALSE if your R version is < 4.0.0 to prevent strings from being interpreted as factors. In addition, the variables Links and Emoji are lists and can contain multiple values per row. For this reason, you need to enclose them with the I() function to be able to put them into a dataframe. You can save your result using the saveRDS() function.