Explore

“The data speaks for itself, but only if you are willing to listen.”
— Nate Silver”

Dr. Jerid Francom

Mar 27, 2024

Overview

• Orientation
• Descriptive analysis
• Unsupervised learning

Process

The Big Picture

Orientation

Exploratory data analysis

Goals

• Discover patterns
• Describe emerging relationships
• Posit hypotheses

When to use

• Literature is scarce
• Gap in knowledge is wide
• New territory

How to use

• Identify, Inspect, Interrogate, Interpret
• Iterative process
  • Update: unit of observation, variables, methods

Descriptive analysis

Frequency analysis: what is it?

Simply put, its counting tokens.

Method	Description
Raw frequency	Number of occurrences of a token within a corpus
Dispersion	Distribution of a token across a corpus
Relative frequency	Proportion of a token in relation to the total number of tokens in a corpus

What can a frequency analysis tell us?

• Varying usage and distributin of tokens can signal:
  • Different populations of speakers
  • Different topics, genres, registers, etc.
  • Cognitive processes (entrenchment, salience)

Things to consider

• Tokenization
• Operationalization
• Zipf distribution

(a) Original

(b) Log-transformed

Figure 1: Zipf distribution (N = 100)

• Define what constitutes a token
• Balancing frequency and importance
• Zipf distribution: a small number of tokens are very frequent, while the majority are rare
• Log-transformed data can help decompress the distribution
• Zipf’s law: the frequency of a token is inversely proportional to its rank

Co-occurrence analysis: what is it?

Identify patterns of association between tokens

Method	Description
n-grams	Sequence of n tokens
Collocation	Tokens that frequently co-occur

What can a co-occurrence analysis tell us?

Patterns of association between tokens can signal:

• Grammatical structures
• Semantic relationships
• Formulaic/ Idomatic expressions

Things to consider

• Operationalization
• Context window
• Measures of association
  • Pointwise mutual information
  • Log-likelihood ratio

• Define what type of association you are interested in
• Context window: the number of tokens before and after the target token
• Measures trade-offs:
  • PMI: strong and frequent associations, sensitive to rare tokens
  • LLR: unexpected and rare associations.

Unsupervised learning

Clustering: what is it?

Bottom-up approach to grouping similar data points

Method	Description
K-means	Partition data into k clusters
Hierarchical clustering	Build a tree of clusters

What can clustering tell us?

• Patterns of similarity/ dissimilarity
• Number of groups
• Homogeneity within groups
• Heterogeneity between groups

(a) K-means

(b) Hierarchical clustering

Figure 2: Examples of K-means and hierarchical clustering

The factoextra package (Kassambara and Mundt 2020) provides functions to visualize clustering results.

Things to consider

• Previous knowledge
• Number of clusters
• Distance metric (Heirarchical clustering)

• Previous knowledge can help determine the number of clusters to compare to
• The elbow method can help determine the optimal number of clusters
• Distance metrics:
  • Cosine similarity: measures the cosine of the angle between two vectors (reduces the effect of magnitude, in Zipfian distribution cases)
  • Euclidean distance: straight-line distance between two points

The proxy package (Meyer and Buchta 2022) provides a dist() function which includes the cosine distance metric which is useful for Zipfian distribution cases.

Dimensionality reduction: what is it?

Operation to reduce the number of variables in a dataset, while preserving as much information as possible

Method	Description
PCA	Linear transformation to reduce dimensionality
t-SNE	Non-linear transformation to visualize high-dimensional data

What can dimensionality reduction tell us?

• How variables are related
• How redundant variables are
• Identify dimensions that explain the most variance

Things to consider

• Interpretability
• Number of dimensions
• Overfitting/ Underfitting

(a) Without PCA

(b) With PCA

Figure 3: K-means clustering with and without PCA

• Interpretability: is the reduced dimensionality still interpretable?
• Number of dimensions: how many dimensions are needed to explain the most variance?
• Overfitting: is the model too complex? Can it generalize to new data? PCA can help reduce overfitting, but may lead to underfitting.

factoextra also provides functions to visualize PCA results.

Word embeddings: what is it?

Use of distributed representations of words in a continuous vector space where words with similar contextual distributions are closer together

Method	Description
Word2Vec	Popular word embedding model
GloVe	Global vectors for word representation

Note: word embedding models are highly contingent on the size of the corpus, the algorithm used, and parameters set.

The word2vec package (Wijffels and Watanabe 2023) provides functions to train and use Word2Vec models.

What can word embeddings tell us?

• Semantic relationships between words
• Grammatical relationships between words
• Analogies between words
• Clustering of words based on context

Things to consider

• Context window
• Number of dimensions
• Pre-trained models vs. training your own model

• Context window: the number of tokens before and after the target token
• Number of dimensions: how many dimensions are needed to explain the most variance?
• Training data: pre-trained models are useful for general tasks, but training your own model can be more specific to your data (size of the corpus, domain-specific vocabulary)

Wrap-up

Final thoughts

• Exploratory data analysis covers a wide range of methods and techniques.
• Of the three approaches we will cover, it requires the most creativity and flexibility.
• The evaluative process is iterative and associative, and the results are often open to interpretation.

References

Kassambara, Alboukadel, and Fabian Mundt. 2020. Factoextra: Extract and Visualize the Results of Multivariate Data Analyses. https://CRAN.R-project.org/package=factoextra.

Meyer, David, and Christian Buchta. 2022. Proxy: Distance and Similarity Measures. https://CRAN.R-project.org/package=proxy.

Wijffels, Jan, and Kohei Watanabe. 2023. Word2vec: Distributed Representations of Words. https://CRAN.R-project.org/package=word2vec.