The repository contains scripts and auxiliary files used in masters HSE work "Representation of minor languages of Russia on the Internet: quantitative description and data analysis".

Counts and prints to stdout different stats about internet text collections: amount of all web-sites, amount of whole-downloaded web-sites, amount of web-pages, amount of tokens, token median per page, whole-downloaded ratio to all web-sites

receives options: -l - to process single folder with jsons -m - to process all lang folders with jsons -u - for folder with lists of whole-downloaded web-sites

For what it is all about, see repo of the project: https://bitbucket.org/LudaLuda/minorlangs/overview (sorry, Russian). Works with text collections similar to web-site collections: http://web-corpora.net/wsgi3/minorlangs/download

Calculated data with other data merged presented here: https://github.com/irinfox/minor_langs_internet_analysis/blob/master/data/langs_all_info_merged.tsv

uses whois-api and ip-api to receive domains registration info Perhaps, works only in linux systems

options: -u - for list with web-sites and langs -a - for list with ambigious web-sites and langs -g - to add geo ip information to the whois-file

you can find all gathered whois information here: https://github.com/irinfox/minor_langs_internet_analysis/blob/master/data/domains_registration_info_with_ip.tsv

drawing script for all registration stats info

Script works in two stages:

1. gets all links from all htmls and save them into json file options -l or -w - to pass folders with htmls outputs folder with json: links_from_htmls

2. gets all connections between different languages options -c (connection mode) and -u (files with url_types from web-site: http://web-corpora.net/wsgi3/minorlangs/download) output:

• files with links info to tsv file (see https://github.com/irinfox/minor_langs_internet_analysis/blob/master/data/links_info.tsv)
• files with graph info to stdout (see https://github.com/irinfox/minor_langs_internet_analysis/blob/master/data/graph_info.txt)
• creates folder link_graphs with plenty of .dot files

NB! Implicitly uses langs_all_info_merged.tsv from data folder in this repo.

Miniscript to draw all dot files saved from get_links_info.py.

Auxiliary scipt to parse Wikipedia page with table of wikipedias in all languages and all information about them

Script with some regularly used functions, such as file reading

There are two ipynb files with Russian comments for data analysis

includes data preparation, analysis of correlations, regression analysis and hierarchical clustering

draws graphs for token median per page and scatter plot of token median per page and web-pages includes comments in Russian