Screenshots
Example Usage
$ courlan --inputfile url-list.txt --outputfile cleaned-urls.txt
$ courlan --help
usage: courlan [-h] -i INPUTFILE -o OUTPUTFILE [-d DISCARDEDFILE] [-v]
[-p PARALLEL] [--strict] [-l LANGUAGE] [-r] [--sample SAMPLE]
[--exclude-max EXCLUDE_MAX] [--exclude-min EXCLUDE_MIN]
Command-line interface for Courlan
options:
-h, --help show this help message and exit
I/O:
Manage input and output
-i INPUTFILE, --inputfile INPUTFILE
name of input file (required)
-o OUTPUTFILE, --outputfile OUTPUTFILE
name of output file (required)
-d DISCARDEDFILE, --discardedfile DISCARDEDFILE
name of file to store discarded URLs (optional)
-v, --verbose increase output verbosity
-p PARALLEL, --parallel PARALLEL
number of parallel processes (not used for sampling)
Filtering:
Configure URL filters
--strict perform more restrictive tests
-l LANGUAGE, --language LANGUAGE
use language filter (ISO 639-1 code)
-r, --redirects check redirects
Sampling:
Use sampling by host, configure sample size
--sa