# README to know how to run weka and the scripts to process the data

# Download the Java version for snowball stemmers at http://snowball.tartarus.org/download.php and extract the Java classes in the folder 'ext'.
# Paste the 'ext' folder in the diretory 'weka-src/src/main/java/weka/core/stemmers/' to compile weka with the snowball stemmer
# The idea is to use the Porter's stemmer for English language that's performs really well for the English language. 
# If you don't want to compile weka with snowball stemmer just put the snowball jar on the classpath

# To compile Weka type:

/begining_of_your_path/stable-3-6/weka$  ant -f build.xml weka-src.jar


# Define the $CLASSPATH to run Weka from a GUI interface or command-line:

CLASSPATH=/begining_of_your_path/stable-3-6/weka/
export CLASSPATH


# To run Weka using a GUI interface type:

/begining_of_your_path/stable-3-6/weka$ java -classpath $CLASSPATH:weka.jar:snowball.jar:libsvm.jar weka.gui.GUIChooser


# Before run the script you have to install the Ruby gems (libraries) needed to run the scripts that are listed in the Gemfile.
# To install these gems type:
/begining_of_your_path/scripts gem install bundler
/begining_of_your_path/scripts bundle install


# To run scripts

/begining_of_your_path/scripts$ ruby arff-document-data-generator.rb tweets-file-name.arff tweets_folder

                 				ruby label_tweets.rb security_tweets.json tweets-folder

                 				ruby data-statistics.rb tweets_folder


data-statistics.rb prints some statistics and the tokens of each class.
The files with the statistics are in the corresponding classes

label_tweets.rb just label tweets to build the tweet set for each phase of classifying process

arff-document-data-generator.rb preprocess the tweets to create a file for Weka tool reading it. 


security_tweets.json: contains the collected tweets that will be labeled
snowball.jar: contains the Porter stemmer that's gonna be used
libsvm.jar: contains the used SVM algorithms
tweets.arff: is the file that contains the pre-processed tweets
tweets-folder represents the directory that contains the set of tweets belonging to a phase of a classification problem.
For example:   tweets/evaluation/part1
			   tweets/evaluation/part2
			   tweets/training-test


# To build preprocessed tweet file for each classifying model run:

/begining_of_your_path/stable-3-6/weka$ bash process_data_normalize_dictionary.sh


process_data_normalize_dictionary.sh do batch filtering[1] for each tweet dataset model designed to make the two sets have the same attributes. So the test set will be able to run against the built classifying models
[1] https://thekandyancode.wordpress.com/tag/batch-filtering/


