Web Spiders

I am going to write on two (2) web spiders that I commonly make use of in this post (there really but 2 are related).

First we have Web Scarab or the "WebScarab web auditing tool". This is a project from the OWASP framework and toolsets. It is designed to help ensure that websites are secure by creating a tool to break them (in simple terms). WebScarab is a Java based framework and web proxy designed for analyzing applications that communicate using the HTTP and HTTPS protocols.
The following list of features and functions taken directly from the OWASP site, and details explicitly why WebScarab is such a critical tool to the Web application tester. The any issue with this tool is that it is not for the faint of heart. You will probably find little value in this tool if you cannot already manually test a site. For those who can however this tool takes Web testing to the next level.

• Fragments - extracts Scripts and HTML comments from HTML pages as they are seen via the proxy, or other plugins
• Proxy - observes traffic between the browser and the web server. The WebScarab proxy is able to observe both HTTP and encrypted HTTPS traffic, by negotiating an SSL connection between WebScarab and the browser instead of simply connecting the browser to the server and allowing an encrypted stream to pass through it. Various proxy plugins have also been developed to allow the operator to control the requests and responses that pass through the proxy.
• Manual intercept - allows the user to modify HTTP and HTTPS requests and responses on the fly, before they reach the server or browser.
• Beanshell - allows for the execution of arbitrarily complex operations on requests and responses. Anything that can be expressed in Java can be executed.
• Reveal hidden fields - sometimes it is easier to modify a hidden field in the page itself, rather than intercepting the request after it has been sent. This plugin simply changes all hidden fields found in HTML pages to text fields, making them visible, and editable.
• Bandwidth simulator - allows the user to emulate a slower network, in order to observe how their website would perform when accessed over, say, a modem.
• Spider - identifies new URLs on the target site, and fetches them on command.
• Manual request - Allows editing and replay of previous requests, or creation of entirely new requests.
• SessionID analysis - collects and analyses a number of cookies (and eventually URL-based parameters too) to visually determine the degree of randomness and unpredictability.
• Scripted - operators can use BeanShell to write a script to create requests and fetch them from the server. The script can then perform some analysis on the responses, with all the power of the WebScarab Request and Response object model to simplify things.
• Parameter fuzzer - performs automated substitution of parameter values that are likely to expose incomplete parameter validation, leading to vulnerabilities like Cross Site Scripting (XSS) and SQL Injection.
• Search - allows the user to craft arbitrary BeanShell expressions to identify conversations that should be shown in the list.
• Compare - calculates the edit distance between the response bodies of the conversations observed, and a selected baseline conversation. The edit distance is "the number of edits required to transform one document into another". For performance reasons, edits are calculated using word tokens, rather than byte by byte.
• SOAP - There is a plugin that parses WSDL, and presents the various functions and the required parameters, allowing them to be edited before being sent to the server.
• Extensions - automates checks for files that were mistakenly left in web server's root directory (e.g. .bak, ~, etc). Checks are performed for both, files and directories (e.g. /app/login.jsp will be checked for /app/login.jsp.bak, /app/login.jsp~, /app.zip, /app.tar.gz, etc). Extensions for files and directories can be edited by user.
• XSS/CRLF - passive analysis plugin that searches for user-controlled data in HTTP response headers and body to identify potential CRLF injection (HTTP response splitting) and reflected cross-site scripting (XSS) vulnerabilities.

The spider function of Web Scarab allows a tester to download an image of the site for offline tests.

On the other hand, Black Widow and Brown Recluse (each by SoftByte Labs) are dedicated spiders.
Black widow provides a searchable spider with the ability to save page and site structure, it also has a RegEx (regular Expression) based search function.
Brown Recluse is a programmable spider. It can be configured to search sites for selected strings or by server type.