Crawpy – However Some other Content material subject material Discovery Device

https://morph3sec.com/crawpy/example.html
https://morph3sec.com/crawpy/example.txt

git clone https://github.com/morph3/crawpy
pip3 arrange -r must haves.txt 
or
python3 -m pip arrange -r must haves.txt

morph3 ➜ crawpy/ [main✗] λ python3 crawpy.py --help
usage: crawpy.py [-h] [-u URL] [-w WORDLIST] [-t THREADS] [-rc RECURSIVE_CODES] [-rp RECURSIVE_PATHS] [-rd RECURSIVE_DEPTH] [-e EXTENSIONS] [-to TIMEOUT] [-follow] [-ac] [-fc FILTER_CODE] [-fs FILTER_SIZE] [-fw FILTER_WORD] [-fl FILTER_LINE] [-k] [-m MAX_RETRY]
[-H HEADERS] [-o OUTPUT_FILE] [-gr] [-l URL_LIST] [-lt LIST_THREADS] [-s] [-X HTTP_METHOD] [-p PROXY_SERVER]
optional arguments:
-h, --help            show this lend a hand message and pass out
-u URL, --url URL     URL
-w WORDLIST, --wordlist WORDLIST
Wordlist
-t THREADS, --threads THREADS
Size of the semaphore pool
-rc RECURSIVE_CODES, --recursive-codes RECURSIVE_CODES
Recursive codes to scan recursively Example: 301,302,307
-rp RECURSIVE_PATHS, --recursive-paths RECURSIVE_PATHS
Recursive paths to scan recursively, please phrase that best given recursive paths will likely be scanned first of all Example: admin,reinforce,js,backup
-rd RECURSIVE_DEPTH, --recursive-depth RECURSIVE_DEPTH
Recursive scan depth Example: 2
-e EXTENSIONS, --extension EXTENSIONS
Add extensions at the end. Seperate them with comas Example: -x .php,.html,.txt
-to TIMEOUT, --timeout TIMEOUT
Timeouts, I like to recommend you to not use this feature because of it is procudes various erros now which I was now not ready to get to the bottom of why
-follow, --follow-redirects
Practice redirects
-ac, --auto-calibrate
Mechanically calibre filter stuff
-fc FILTER_CODE, --filter-code FILTER_CODE
Filter status code
-fs FILTER_SIZE, --filter-size FILTER_SIZE
Filter duration
-fw FILTER_WORD, --filter-wo   rd FILTER_WORD
Filter words
-fl FILTER_LINE, --filter-line FILTER_LINE
Filter line
-k, --ignore-ssl      Overlook about untrusted SSL certificate
-m MAX_RETRY, --max-retry MAX_RETRY
Max retry
-H HEADERS, --headers HEADERS
Headers, you are able to set the flag a few circumstances.As an example: -H "X-Forwarded-For: 127.0.0.1", -H "Host: foobar"
-o OUTPUT_FILE, --output OUTPUT_FILE
Output folder
-gr, --generate-report
If you want to have crawpy to generate a document, default path is crawpy/tales/<url>.txt
-l URL_LIST, --list URL_LIST
Takes a list of urls as input and runs crawpy on by the use of multiprocessing -l ./urls.txt
-lt LIST_THREADS, --list-threads LIST_THREADS
Number of threads for running crawpy parallely when running with document of urls
-s, --silent          Make crawpy now not produce output
-X HTTP_METHOD, --http-method HTTP_METHOD
HTTP request elements
-p PROXY_SERVER, --proxy PROXY_SERVER
Proxy server, ex: 'http://127.0.0.1:8080'

python3 crawpy.py -u https://facebook.com/FUZZ -w ./not unusual.txt  -k -ac  -e .php,.html
python3 crawpy.py -u https://google.com/FUZZ -w ./not unusual.txt  -k -fw 9,83 -r 301,302 -rd 2
python3 crawpy.py -u https://morph3sec.com/FUZZ -w ./not unusual.txt -e .php,.html -t 20 -ac -k
python3 crawpy.py -u https://google.com/FUZZ -w ./not unusual.txt  -ac -gr
python3 crawpy.py -u https://google.com/FUZZ -w ./not unusual.txt  -ac -gr -o /tmp/check out.txt
sudo python3 crawpy.py -l urls.txt -lt 20 -gr -w ./not unusual.txt -t 20 -o custom_reports -k -ac -s
python3 crawpy.py -u https://google.com/FUZZ -w ./not unusual.txt -ac -gr -rd 1 -rc 302,301 -rp admin,backup,reinforce -k