UrlBuster – Linux tool to find Web Hidden Files or Directories Finder

Hidden files and directories on the target server can contain some sensitive and crucial information about the target. This revealed information can also compromise the security of the application. To find these directories and files, we need an automated approach as manual testing would make a headache to the tester. UrlBuster is an automated tool developed in the python language, which digs the hidden directories and files from the target server. UrlBuster supports using a custom word list for performing brute-forcing of directories and files. It supports the No TLS feature to remain unknown or hide your identity. UrlBuster tool is open-source and free to use the tool.

Note: Make Sure You have Python Installed on your System, as this is a python-based tool. Click to check the Installation process: Python Installation Steps on Linux

Installation of UrlBuster Tool on Kali Linux

Step 1: Check whether Python Environment is Established or not, use the following command.

python3

Step 2: Install the Tool using the Pip, use the following command.

sudo pip3 install urlbuster 

Step 3: Check the help section of the tool using the following command.

urlbuster --help

Working with UrlBuster Tool on Kali Linux

Example 1: Fuzzing with Wordlist/Basic Usage

urlbuster -W /usr/share/wordlists/dirb/common.txt http://geeksforgeeks.org/

1. In this example, We will be fuzzing directories by performing brute-forcing. We have specified the word list using -W tag.

2. In the below Screenshot, We have got the results or the directories of our target domain geeksforgeeks.org

Example 2: No TLS Verification

urlbuster -k -W /usr/share/wordlists/dirb/common.txt http://geeksforgeeks.org/

1. In this example, We are bypassing the TLS Verification by using the -k tag.

2. In the below Screenshot, We have got the results of our fuzz. We can get the information anonymously without TLS verification.

Example 3: New Connection

urlbuster -n -k -W /usr/share/wordlists/dirb/common.txt http://geeksforgeeks.org/

In this example, We are building a new connection with the target domain by using the -n tag. This will help to remain anonymous because of a new connection for every request and if not specified persistent HTTP connection will be used for all requests.

Example 4: Verbose

urlbuster -n -k -W /usr/share/wordlists/dirb/common.txt http://geeksforgeeks.org/ -v

In this example, We are displaying the results in a more detailed manner. We have used the -v tag for verbose mode.

Example 5: HTTP status code

urlbuster -n -k -W /usr/share/wordlists/dirb/common.txt http://geeksforgeeks.org/ –code 200 

In this example, We only want to see the specific HTTP response or status code then we go with it. We can add more response codes by giving the space.

2. In the below Screenshot, We will be getting only the results that contain status code 200.

Example 6: Change Header

urlbuster -n -k -W /usr/share/wordlists/dirb/common.txt http://geeksforgeeks.org/ –header key:ubuntu

1. In this example, we are setting a custom HTTP header string that will be sent to the server and the target domain will not identify the exact details.

2. In the below Screenshot, We have specified the custom header string.

Example 7: Method

urlbuster -n -k -W /usr/share/wordlists/dirb/common.txt http://geeksforgeeks.org/ –method POST

1. In this example, We are changing the method of Request form GET to POST method. We have used –method tag to specify the method.

2. In the below Screenshot, We have to change the Request method to POST.

Example 8: Proxy through Burpsuite

urlbuster -W /usr/share/wordlists/dirb/common.txt –proxy ‘http://127.0.0.1:80’ http://geeksforgeeks.org/

In this example, We will use Proxy through Burpsuite

Example 9: Save results to file

urlbuster -W /usr/share/wordlists/dirb/common.txt –proxy ‘http://127.0.0.1:80’ http://geeksforgeeks.org/ –output out.txt

http://127.0.0.1:80

In this example, We will Save Results into a file.

Example 10: Scan behind Basic Auth

urlbuster -W /usr/share/wordlists/dirb/common.txt –proxy ‘http://127.0.0.1:80’ http://geeksforgeeks.org/  –auth-basic ‘user:pass’

In this example, We will Scan behind Basic Auth.

Example 11: Use session cookie

urlbuster -W /usr/share/wordlists/dirb/common.txt –proxy ‘http://127.0.0.1:80’ http://geeksforgeeks.org/ –cookie ‘authtoken=4ee5d467598030eaf1bed96473ed2d99’

http://127.0.0.1:80

In this example, We will use the session cookie while scanning.

Example 12: Find files in the root directory

urlbuster -W /usr/share/wordlists/dirb/common.txt http://geeksforgeeks.org/ –code 200 301 302 –ext .zip .tar .tar.gz .gz .rar

In this Example, We will Find files in the root directory.

Example 13: Find files in subdirectory

 urlbuster -W /usr/share/wordlists/dirb/common.txt http://geeksforgeeks.org/write –code 200 301 302 –ext .zip .tar .tar.gz .gz .rar

In this example, We will Find files in a subdirectory.

Example 14: Brute force query parameter

urlbuster -W /usr/share/wordlists/dirb/common.txt https://www.geeksforgeeks.org/fundamentals-of-algorithms/?ref=shm

In this example, We will Brute force query parameter.

Example 15: Brute force POST requests

urlbuster -W /usr/share/wordlists/dirb/common.txt https://www.geeksforgeeks.org/fundamentals-of-algorithms/?ref=shm –code 200 301 302 –method POST

In this example, We will Brute force POST requests

Example 16: Brute force mutated POST requests

urlbuster -W /usr/share/wordlists/dirb/common.txt https://www.geeksforgeeks.org/fundamentals-of-algorithms/?ref=shm  –code 200 301 302 –method POST  –mpayload ‘user=somename1’

In this example, We will Brute force mutated POST requests.

Example 17: User-agent SQL injections

urlbuster -W /usr/share/wordlists/dirb/common.txt https://www.geeksforgeeks.org/fundamentals-of-algorithms/?ref=shm –code 200 301 302 –method POST –mheader “User-Agent: ;”

In this example, We will use User-agent SQL injections.

Example 18: Find potential vhosts

urlbuster -W /usr/share/wordlists/dirb/common.txt https://www.geeksforgeeks.org/fundamentals-of-algorithms/?ref=shm –code 200 301 302 –method POST –mheader “Host: internal1.lan”

In this example, We will Find potential vhosts.