Option 1: Data is in a dataframe

The package comes with sample data. For this demonstration, we use dev_coop. The raw data looks like this:

We will look at question 11_3 (responses to ‘’Jatka lausetta: Maailman kolme suurinta ongelmaa ovat… (Avokysymys)’) as our open-ended survey question. We also want to include our survey weights (in ‘paino’ column) and bring in the gender and region columns so we can use these values to compare groups.

The main function here is fst_prepare()

# FUNCTION DEFINITION
fst_prepare <- function(data,
                        question,
                        id,
                        model = "ftb",
                        stopword_list = "nltk",
                        language = "fi"
                        weights = NULL,
                        add_cols = NULL,
                        manual = FALSE,
                        manual_list = "")

We can run the function as follows:

df <- fst_prepare(data = dev_coop,
                  question = 'q11_3', 
                  id = 'fsd_id', 
                  weights = 'paino',
                  add_cols = c('gender', 'region')
                  )

Summary of components

• data is the dataframe of interest. In this case, we are using data that comes with the package called “dev_coop”.
• The question is the name of the column in your data which contains the open-ended survey question. In this example, we’re considering “q11_3”
• id is the id column in our data, which is “fsd_id”
• The function also requires a language model available for udpipe, in this case we are using the default Finnish Treebank, model = "ftb". (There are two options for Finnish language model; the other option is the Turku Dependency Treebank “tdt”.)
• By default we will remove stopwords from the “nltk” stopword_list in this example. (To find the relevant lists of Finnish stopwords, you can run the fst_find_stopwords() function.) Punctuation is also removed from the data whenever stopwords are removed.
• The default language is “fi”. This should be the two-letter ISO code for the language for the stopword list.
• Optionally, you can add a weights column in your formatted data. Our weights are stored in the raw data as “paino”.
• Optionally, you can add other columns to your formatted data (for use in comparison functions). We include our “gender” and “region” columns for this reason
• The results in CoNLL-U format are stored in the local environment as df.
• (manual and manual_list are used if you want to provide your own list of stopwords to remove from the data.)

The formatted data looks like this:

Option 2: Data is in a svydesign object

The other option is to get your data from a svydesign object from the survey package. The survey package is a popular package used for analysing surveys.

The main function here is fst_prepare_svydesign()

# FUNCTION DEFINITION
fst_prepare_svydesign <- function(svydesign,
                                  question,
                                  id,
                                  model = "ftb",
                                  stopword_list = "nltk",
                                  language = "fi"
                                  use_weights = TRUE,
                                  add_cols = NULL,
                                  manual = FALSE,
                                  manual_list = "") 

We can run the function as follows:

df2 <- fst_prepare_svydesign(svydesign = svy_dev,
                            question = 'q11_3', 
                            id = 'fsd_id', 
                            use_weights = TRUE,
                            add_cols = c('gender', 'region')
                            )

The only differences between the previous function and this one are:

• svydesign is your svydesign object. In this case, we have one called “svy_dev”
• The svydesign object has a component called “prob” which contains the inverse of the weights. Therefore, we use these by setting use_weights = TRUE

The formatted data looks like this (should look very similar to the above formatted data!):

Summary Tables

First, let’s create some summaries using fst_summarise, fst_pos and fst_length_summary

These functions are defined as follows:

# FUNCTION DEFINITIONS
fst_summarise <- function(data, 
                          desc = "All respondents") 

fst_pos <- function(data) 
  
fst_length_summary <- function(data,
                               desc = "All respondents",
                               incl_sentences = TRUE) 

Summary of components

• data is the formatted data.
• desc is an optional name for the responses summarised, if not provided it will default to “All respondents”.
• incl_sentences is an optional boolean for whether to also summarise sentence length (in addition to word length), if not provided it will default to TRUE.

Hence, these functions are run for our sample data as follows:

fst_summarise(df)

##     Description Respondents No Response Proportion Total Words Unique Words
## 1 All responses         920          25       0.97        4192         1132
##   Unique Lemmas
## 1           994

fst_pos(df)

##     UPOS                  UPOS_Name Count Proportion
## 1    ADJ                  adjective   389      0.093
## 2    ADP                 adposition    24      0.006
## 3    ADV                     adverb    64      0.015
## 4    AUX                  auxiliary     3      0.001
## 5  CCONJ   coordinating conjunction     3      0.001
## 6    DET                 determiner    28      0.007
## 7   INTJ               interjection     2      0.000
## 8   NOUN                       noun  3286      0.789
## 9    NUM                    numeral     5      0.001
## 10  PART                   particle    29      0.007
## 11  PRON                    pronoun    12      0.003
## 12 PROPN                proper noun    31      0.007
## 13 PUNCT                punctuation     0      0.000
## 14 SCONJ  subordinating conjunction     0      0.000
## 15   SYM                     symbol     1      0.000
## 16  VERB                       verb   278      0.067
## 17     X                      other    12      0.003

fst_length_summary(df)

## # A tibble: 2 × 8
##   Description              Respondents  Mean Minimum    Q1 Median    Q3 Maximum
##   <chr>                          <int> <dbl>   <int> <dbl>  <dbl> <dbl>   <int>
## 1 All responses- Words             920  5.52       1     4      5     6      32
## 2 All responses- Sentences         920  1.01       1     1      1     1       3

Identification of frequent words and phrases

# FUNCTION DEFINITION
fst_wordcloud <- function(data,
                          pos_filter = NULL,
                          max = 100,
                          use_svydesign_weights = FALSE,
                          id = "",
                          svydesign = NULL,
                          use_column_weights = FALSE)

fst_wordcloud(df)

# We can only get weights from svydesign if they are NOT already in our formatted data. Hence we remove them for this demonstration!
df2$weight <- NULL
fst_wordcloud(df2, 
              pos_filter = c("NOUN", "VERB", "ADJ", "ADV"),
              max=100, 
              use_svydesign_weights = TRUE, 
              id = 'fsd_id', 
              svydesign = svy_dev)

# FUNCTION DEFINITIONS
fst_freq <- function(data,
                     number = 10,
                     norm = NULL,
                     pos_filter = NULL,
                     strict = TRUE,
                     name = NULL,
                     use_svydesign_weights = FALSE,
                     id = "",
                     svydesign = NULL,
                     use_column_weights = FALSE)
  
fst_ngrams <- function(data,
                       number = 10,
                       ngrams = 1,
                       norm = NULL,
                       pos_filter = NULL,
                       strict = TRUE,
                       name = NULL,
                       use_svydesign_weights = FALSE,
                       id = "",
                       svydesign = NULL,
                       use_column_weights = FALSE)

fst_freq(df)

fst_ngrams(df, 
           number = 9, 
           ngrams = 2, 
           strict = FALSE,
           use_column_weights = TRUE)

fst_freq(df,
         number = 5, 
         strict = FALSE,)

fst_freq_table(df, number = 15, strict = FALSE)

##                 words occurrence
## 1             köyhyys        258
## 2           nälänhätä        239
## 3                sota        231
## 4      ilmastonmuutos        141
## 5               puute        117
## 6             ihminen        105
## 7                vesi         98
## 8        epätasa-arvo         87
## 9              ahneus         84
## 10              nälkä         81
## 11             puhdas         75
## 12            sairaus         59
## 13          itsekkyys         58
## 14       väestönkasvu         48
## 15 välinpitämättömyys         47

1. Child Barometer Data

• Source: FSD3134 Lapsibarometri 2016
• Question: q7 ‘Kertoisitko, mitä sinun mielestäsi kiusaaminen on? (Avokysymys)’
• Licence: (A) openly available for all users without registration (CC BY 4.0).
• Link to Data: https://urn.fi/urn:nbn:fi:fsd:T-FSD3134

2. Development Cooperation Data

• Source: FSD2821 Nuorten ajatuksia kehitysyhteistyöstä 2012
• Questions: q11_1 ‘Jatka lausetta: Kehitysmaa on maa, jossa… (Avokysymys)’, q11_2 ‘Jatka lausetta: Kehitysyhteistyö on toimintaa, jossa… (Avokysymys)’, q11_3 ‘Jatka lausetta: Maailman kolme suurinta ongelmaa ovat… (Avokysymys)’
• Licence: (A) openly available for all users without registration (CC BY 4.0).
• Link to Data: https://urn.fi/urn:nbn:fi:fsd:T-FSD2821

3. Patient Joe (open-ended question)

• Source: GESIS – Leibniz Institute for the Social Sciences
• Open-ended question: ‘Joe’s doctor told him that he would need to return in two weeks to find out whether or not his condition had improved. But when Joe asked the receptionist for an appointment, he was told that it would be over a month before the next available appointment. What should Joe do?’
• Licence: CC BY 4.0: Attribution
• Link to Data: https://doi.org/10.7802/2474