ITC565 Developer's Blog

Just because

2009-08-25T09:45:00.002+10:00

Look, just as I am leaving this blog and am not using it does not mean that I will allow it to be a Splog.

To the SCUM who keep (attempted) posting crud regarding their crap products - I will be blocking and filtering ALL comments as usual.

Sploggers are LOSERS!

Where to now?

2009-07-27T11:44:00.002+10:00

The exam is looming and the assignments are submitted, so it is back to the former blog.

I will be adding a number of posts on both the SANS forensic Blog as well as my own in the next few days. I am leaving this material live on the web (even after the close of the session) and I hope that it is of use.

Assessment item 2

2009-07-27T11:20:00.002+10:00

This page links to the various sub-pages and posts associated with Assignment No.2.

The second Assignment comprises of 14 exercises:

Ruby on Rails Workshops 5 to 8

And the second Elevator Pitch.

Elevator pitch 2 on progress report for assessment item 2

Where needed, there are also additions to Assignment 1.

CASE (Computer Aided Software Engineering) Tools

2009-07-27T10:38:00.002+10:00

Case tools can be a great aid to auditing database systems. CASE or Computer Assisted Software Engineering tools not only help in the development of software and database structures but can be used to reverse engineer existing databases and check them against a predefined schema. There are a variety of both open source and commercial CASE tools. In this chapter we’ll be looking at Xcase (http://www.xcase.com/).

Many commercial databases can run into the gigabyte or terabyte in size. Standard command line SQL coding is unlikely to find all of the intricate relationships between these tables, stored procedures and other database functions. A CASE tool on the other hand can reverse engineer existing databases to produce diagrams that represent the database. These can first of all be compared with existing schema diagrams to ensure that the database matches the architecture that it is originally built from and to be able to quickly zoom in on selected areas.

Visual objects, colors and better diagrams may all be introduced to further enhance the auditor’s capacity to analyze the structure. Reverse engineering a database will enable the auditor to find out the various structures that have been created within the database. Some of these include:

The indexes,
Fields,
Relationships,
Sub-categories,
Views,
Connections,
Primary keys and alternate keys,
Triggers,
Constraints,
Procedures and functions,
Rules,
Table space and storage details associated with the database,
Sequences used and finally the entities within the database.

Each of the tables will also display detailed information concerning the structure of each of the fields that may be viewed at a single glance. In large databases a graphical view is probably the only method that will adequately determine if relationships between different tables and functions within a database actually meet the requirements. It may be possible in smaller databases to determine the referential integrity constraints between different fields, but in a larger database containing thousands of tables there is no way to do this in a simple manner using manual techniques.
Fig- Display database schema.
When reviewing the database design, it is not just security functions such as cross site scripting and sequel injection that need to be considered. Relationships between various entities and the rights and associated privileges that are associated with various tables and roles also need to be considered. The CASE tools allow us to visualize the most important security features associated with a database. These are:

Schemas restrict the views of the database for users,
Domains, assertions, checks and other integrity controls defined as database objects which may be enforced using the DBMS in the process of database queries and updates,
Authorization rules. These are rules which identify the users and roles associated with the database and may be used to restrict the actions that a user can take against any of the database features such as tables or individual fields,
Authentication schemes. These are schemes which can be used to identify users attempting to gain access to the database or individual features within the database.
User defined procedures which may define constraints or limitations on the use of the database,
Encryption processes. Many compliance regimes call for the encryption of selected data on the database. Most modern databases include encryption processes that can be used to ensure that the data is protected.
Other features such as backup, check point capabilities and journaling help to ensure recovery processes for the database. These controls aid in database availability and integrity, two of the three legs of security.

CASE tools also contain other functions that are useful when auditing a database. One function that is extremely useful is model comparison.
Fig - Reverse Engineer existing databases into presentation quality diagrams in minutes.

Case tools allow the developer to:

Present clear data models at various levels of detail using visual objects, colors and embedded diagrams to organize database schemas,
Synchronize models with the database,
Compare a baseline model to the actual database (or to another model),

Case tools can generate code automatically and also store this for review and baselining. This includes:

DDL Code to build and change the database structure
Triggers and Stored Procedures to safeguard data integrity
Views and Queries to extract data

The auditor can also document the database design using multiple reporting options. This allows for the printing of diagrams and reports and the addition of comments to the reports and user defined attributes to the model.

Data management features allow the auditor to validate the data in the database being reviewed against the business rules and constraints defined in the model and generate detailed integrity reports. This can be extended further to access and edit the data relationally using automatic parent/child browsers and lookups and then to locate faulty data subsets using automatically generated SQL statements. These provide valuable sources of errors and help in database maintenance – making the audit all the more valuable.

Model comparison involves comparing the model of the database with the actual database on the system. This can be used to ensure change control or to ensure that no unauthorized changes have been made for other purposes. To do this, a baseline of the database structure will be taken at some point in time. At a later time the database could be reverse engineered to create another model and these two models could be compared. Any differences, variations or discrepancies between these would represent a change. Any changes should be authorized changes and if not, should be investigated. Many of the tools also have functions that provide detailed reports of all discrepancies.

Many modern databases run into the terabytes and contain tens of thousands of tables. A baseline and automated report of any differences, variations or discrepancies makes the job of auditing change on these databases much simpler. Triggers and stored procedures can be stored within the CASE tool itself. These can be used to safeguard data integrity. Selected areas within the database can be set up such as honeytoken styled fields or views that can be checked against a hash at different times to ensure that no-one has altered any of these areas of the database. Further in database tables it should not change. Tables of hashes may be maintained and validated using the offline model that has stored these hash functions already. Any variation would be reported in the discrepancy report.

Next the capability to create a complex ERD or Entity Relationship Diagram in itself adds value to the audit. Many organizations do not have a detailed structure of the database and these are grown organically over time with many of the original designers having left the organization. In this event it is not uncommon for the organization to have no idea about the various tables that they have on their own database.

Another benefit of CASE tools is their ability to migrate data. CASE tools have the ability to create detailed SQL statements and to replicate through reverse engineering the data structures. They can then migrate these data structures to a separate database. This is useful as the data can be copied to another system. That system may be used to interrogate tables without fear of damaging the data. In particular the data that has migrated to the tables does not need to be the actual data, meaning that the auditor does not have access to sensitive information but will know the defenses and protections associated with the database. This is useful as the auditor can then perform complex interrogations of the database that may result in damage to the database if it was running on the large system. This provides a capability for the auditor to validate the data in the database against the business rules and constraints that have been defined by the models and generate detailed integrity reports. This capability gives an organization advanced tools that will help them locate faulty data subsets through the use of automatically generated SQL statements.

UML and Mapping Processes

2009-07-27T10:29:00.001+10:00

Unified Modeling Language (UML) is a visual representation language designed for the purpose of modeling and communicating the information contained within systems. To do this it uses a series of diagrams and supporting text.

It can provide details of many process fields such as the following:

Actors, examples could include a manager leading a team executing a project and staff members on the project team.
The various processes that occur.
Relationships between actors and entities.

Unified
In UML, unified came about due to the Object Management Group (OMG) and Rational Software Corporation coming together to create an industry standard for engineering practices. This was a desire to create a common language.

Model
A model is a depiction of a subject. A model is used to encapsulate a set of ideas (called abstractions) concerning a subject. A model provides a simple means to create a common understanding amongst different team members and other individuals. This helps create an understanding of the requirements of the system and to communicate the impact of changes that will occur to the system through development and use.

The creation of a model should be done in stages. An attempt to create a model all in one go is likely to become overwhelming. This may be possible and small systems, but large systems with many thousands of tables are beyond the human capacity to comprehend at once.

When modeling, good practice dictates that the auditor will capture the relevant information that is required to gain an understanding of the problem at hand. This information may then be used to solve problems are issues that have arisen and will aid in the recommendation of a solution. It is also necessary to exclude information that is not relevant to the task at hand. It is easy to be waylaid by immaterial facts that can in no way lead to a change in the system or are not related to the scope of a project.

In order to effectively manage the overall complexity involved within the audit of complex systems such as mainframes, models are an effective tool to achieve our goal. This process is best completed through:

Managing the abstractions that make up the model,
Including enough detail to understand the abstraction but not so much as to sidetrack the audit,
Exclude irrelevant information, and
Work with multiple teams to ensure that the model is relevant.

Language
A language enables both people and systems to communicate about a subject. The subject incorporates the requirements and the system with respect to system development and audit. Language simplifies the process of communicating between individual team members and allows for the successful completion of the project.

Languages are not always composed of words. In fact, complex abstractions such as mathematics are in fact languages.

UML is formally defined by its creators as a language for specifying, visualizing, constructing, and documenting the artifacts of a system-intensive process. This is a system-intensive process used as an approach that centers on a system. It includes the various stages used to both produce and maintain a system. This is based on the requirements needed by the system. The specification includes the creation of a model describing the system. This model simplifies the analysis of the system and allows even complex systems to be audited within a reasonable timeframe and scope.

This process involves visualization through the use of diagrams designed to render the model into a simple form so that it can be communicated. This diagram is then an expression of the system. It could be likened to a blueprint for a building. Ideally, his blueprint is designed before the building, but like many system design projects, development of a model or blueprint has either been excluded or lost. The subsequent creation of this model through audit captures a baseline that can be used not only to understand the process at hand but also for use in future reviews and assessments. Documenting these systems captures the knowledge and requirements associated with the system.

UML and Processes
UML is not a process, it as a tool to capture processes and system design. A process relates a series of stages that are illustrated through the use of a methodology in order to decipher an issue. It then enables the development of a system that is designed to satisfy the requirements of a system owner or users.

Method addresses the following stages of the development process:

Requirements or information gathering,
analysis, and
design.

Methodology addresses the entire development process starting with the requirements or information gathering through to the system being made live.

The distinct means of collecting and using requirements, analyzing requirements and finally designing a system are the techniques utilized. Artifacts are the “work products” produced and used within a process. These include the documentation and the actual system.

Each classification of UML diagram is known as a modeling technique.
The use of a UML diagram (as depicted in the figure above) can greatly simplify the audit process for complex systems.

Further information about UML
The following sites are the principal sources for information about the UML standard:

The Object Management Group (OMG),

o http://www.omg.org and http://www.omg.org/uml

Rational Software Corporation (IBM),

o http://www.rational.com and http://www.rational.com/uml

The subsequent sites present information concerning the next major change to the UML (the OCL) and a variety of other information on the subject:

The Object Constraint Language (OCL),

o http://www.klasse.nl/ocl/index.html

The UML Forum is a virtual community concerning the UML,

o http://www.uml-forum.com

The Cetus Team provides UML tools, methodologies and processes,

o http://www.cetus-links.org

Code Reviews and Testing Third Party Software

2009-07-27T10:23:00.002+10:00

An in-depth study of software audit is beyond the scope of this book, it is however necessary to touch on the subject. In earlier chapters testing methodologies that also relate to software have been described. These range from the black box test commonly used when code is unavailable (such as in the case of third-party software reviews and reviews package software) through to white box and crystal box assessments. In the latter, all code is available and tested.

It is not essential that the auditor understands the intricacies of coding. Rather, it is sufficient to understand how the various testing approaches function and to have sufficient understanding to be able to work with the test engineer who has designed the test cases associated with software in order to be able to understand their work. In particular, the auditor should be able to understand the reports produced by the test engineer.

We shall quickly rehash the types of software audit and review before going further. At the extremes these are:
Black box testing
Black box software testing does not require any understanding of internal behavior. No access to code is available, but rather the response to input is validated. UML diagrams may be available in some instances and in this case a test of functionality will be matched to the functional requirements in the specification. In any event, input will be matched to output to test for expected or unexpected behavior. Some of the various testing methods include:

Equivalence partitioning,
Boundary value analysis,
All-pairs testing,
Fuzzing,
Model-based testing, and
Traceability matrix.

White box testing
This type of testing includes access to the internal data structures. At the extreme (crystal box tests), the tester has access to all code, algorithms and design notes. White box testing will include tests to ensure predefined criteria have been met. Some examples of this form and testing include:

Static code testing,
Mutation testing,
Completeness testing,
Fault injection testing, and
Lexical code analysis.

Testing in Combination
The most effective means of testing software comes from combination of methods being deployed together. Unfortunately, access to code is not always available. In cases of packaged software and many third-party products, access to the code is restricted. Access to code is also effective in increasing the capabilities of the traditional black box test (commonly called a grey box test when code is available to conduct the test using black box test methods).
Correcting a software problem after the event is far more expensive than stopping it before it goes into production release. It is often stated that post-release fixes are in the order of hundreds of times more expensive to fix then when compared to correcting the issue in code and requirements reviews.

When auditing software is necessary to consider the following aspects of development associated with the code:

Software Quality

Correctness,
Completeness,
Integrity,

Capability,
Reliability,
Efficiency,
Portability,
Maintainability,
Compatibility, and
Usability.

Test engineers will generally develop metrics to report on each of these aspects of software development.

The various Levels of Testing
Unit testing
Unit testing focuses on individual software modules (the components of the software). Each module is tested individually in order to validate the software implementation component by component. An example would be the testing of individual classes associated within an object-oriented development environment.

Integration testing
Integration testing is designed to uncover defects in the interfaces and interaction amid the integrated software modules. This form of testing starts with individual modules and joins them to form progressively larger associative groups. Each phase works on larger groupings until the software architecture is tested as an entire system.

Acceptance testing
Acceptance testing is conducted by the end-user. The goal is to decide whether or not to accept the final software product. Acceptance testing may be conducted between development phases.

Regression testing
Regression testing is a process where a previously conducted test is a rerun on the software. This type of testing is conducted in order to ensure that prior defects have not been reintroduced or regressed into the code. This type of testing is frequently automated.
Some specific types of regression testing include sanity testing (this is a check for unexpected and unforeseen behavior) and smoke testing (which is a test to ensure that the product provides basic functionality).

Test Cycles
There are many ways of engineering software. Each of these comes with its own test methodologies. One of the more common ones is the Software Development Life Cycle (SDLC). Some of the common foes is involved with testing include many phases of the project that are analogous to many other audit processes.

Requirements analysis
The first stage of testing generally starts with the creation of a document detailing what is necessary. In this phase both developers and testers will work together to determine what tests may be conducted.

Test planning
This phase includes the creation of a strategy and the scope of the testing. Like an audit, system testing should be conducted as a project. Some areas to consider include:

The creation of a test strategy,
The formulation of a test plan, and
The creation of a test bed or other testing system.

Test development
The development phase of testing involves the creation of a number of test procedures based on the requirements derived in the preceding stages. Some of the steps involved with this phase of testing include:

The development of test procedures,
The creation test scenarios,
Creating test cases and populating simulated data,
The creation of test programs and scripts and possibly the sourcing of third-party testing software (such as the static analysis platforms by Fortify).

Test execution
The test execution phase involves the actual testing of the software based on the processors decided above. Any errors or defects in the code would then be reported to the development team.

Test reporting
Test metrics that were developed in the preceding stages coupled with data concerning errors and defects and possibly recommendations for improvement. This will also include recommendations whether the software needs further testing before being released.
Retesting the defects

Defects may be the result of either errors in the code or the test process itself. It is necessary to ensure that any defects that are a result of the testing process are rectified. Defects may or may not be corrected. Many defects do not have a security related consequence and could be left for future software versions.

Exercise 25 - Q2

2009-07-27T09:47:00.002+10:00

Visit an airline Web site and search for information on WAP or SMS access to booking airline services. Do the same for WAP or SMS services in banking. How do both industries compare?

I have used the NAB and Westpac SMS services. I have to stated that each sucks.

I also use the QANTAS SMS service, there is some use in this. When waiting for a flight, having the data pushed is a good idea.

Banking, well this was and is a waste of time, a solution looking for an issue. It adds little security as SMS is simple to spoof etc and is impracticable.

WAP was dead before it was birthed. This still born technology was a cut down - yet slower - version of the web. As phones and mobile devices become more powerful (and I have been using full browsers on phones for 12 years now) complete rich browsers will become more common on the phone itself. Windows Mobile, iPhone etc all have moved towards the richer model and WAP is slowly fading into the obscurity it deserverd.

Location based services

2009-07-27T09:45:00.001+10:00

Location based services are used to market solutions (such as restaurants) that are generally restricted in a geographically centered way.

They aim to let people know that there is a particular service offered in their current area.

Database Triggers

2009-07-27T09:36:00.002+10:00

Triggers are a method that can be used to add some logic to the database system. When a set action occurs (such as the arrival of a certain type of data), a process can be automatically started. Some of the common database triggers include the following.

Check Triggers
Database triggers are procedural code that is automatically executed in reaction to selected events on a particular table, row or field in a database. The auditor should check that these are used and where. Triggers need to be set to fire when events that are defined in policy occur.

System triggers
System triggers allow the activation of controls that start when system events take place. These events can include:

The start- up and shutdown of the database,
Logon and logoff from users,
Privileged access, and
The creation, altering and dropping of schema objects.

Using autonomous transactions also allows a log to be written for the above system events. The audit should check what (if any) systems triggers exist and ensure that these are aligned with the policy of the organization.

Update, delete, and insert triggers
Defense in depth requires an understanding of the users' actions at multiple levels. This is not just access to the database, but access at the detailed row level for selected events and where there is sensitive data. Database triggers need to be written to capture changes at the column and row level.

Where data is extremely sensitive and any and all changes must be recorded, the database can be configured to write entire rows of data detailing a change to the data (who, what, where and why). This can be done both ahead of and subsequent to the modification of data being made with a write of information to a log table in the database and to an alternate location. This class of logging is extremely resource intensive. It requires that at least as many extra records are written and stored as the planned change (and at times more).

The one flaw in this technique is an inability to capture read access to a file using normal database triggers.

Oracle (as an example) breaks audit into three areas that can be used for logging and in creating triggers:

Statement auditing (CREATE TABLE or CREATE SESSION),
Privilege auditing (ALTER USER), and
Object level auditing (SELECT TABLE).

These inbuilt levels of auditing can provide the auditor with a rich source of evidence in the form of logs.

Triggers are also commonly used as a form of database control. They can be used to trigger the execution of other procedures. For instance, integrity controls may be used to place an entry into a log to record access to tables. In this way user access may be recorded. It is possible to record information across different tables.

Database triggers are also effective in adding security controls to a database. A trigger can include an event, condition and action. Triggers may be more complex than an assertion but will allow the database to automatically prohibit inappropriate actions, automatically start handling procedures using stored procedures or other processes or write a row to a log file. This may be used to reflect information about the user and transaction that has been created. This log may then be displayed in a format that can be read by humans or using automated procedures and tools. Like any stored procedure domains and triggers can be used to enforce controls for all users and all database activities.

Splogs

2009-07-27T09:18:00.001+10:00

Sploggers are one of the most common sources of plagiarism on the Internet. A small number of resolute and capable Sploggers can steal content from thousands of different sites, scraping RSS feeds from them and stealing the content. The change is that many “black hats” have taken up the art. The profit motivation of Sploggers is obvious, how they make a profit is less perceptible.

Splogs were certainly not intended for humans to view. Human-visited Splogs are high risk with little prospective gain. Rather, Splogs consist of links to other sites which are more often than not long junk domains burdened with keywords and metatags. The idea is to have search engines pick up their site. A Splogger’s site will typically consist of nothing but keywords and metatags loaded into the HTTP header with a small amount of random text (usually copied from another site) and numerous diverse groups of text ads arranged to look alternatively like search results or regular links. When time the site is ready to be used, over 90% of the site consists of ads from Adsense or a comparable service.

With sufficient spam links to the site, it is anticipated that the Splogger will rank highly in the search rankings and be besieged by visitors to those sites who they expect will click on the links (Note: According to most SEO experts and my own research, this does NOT work. You can only expedite getting listed, not drastically improve your ranking, thus hundreds of junk posts are a waste). It is hoped that the targeted visitors will subsequently click on the ads, either out of curiosity or due to the mistaken belief that they are regular links. Splogging is a classic example of black hat search engine optimisation (SEO) that merely involves extensive plagiarism to make it work.

The expression “splog” was popularized in August 2005 when it was termed publicly by Mark Cuban. The name was used a sporadically prior to this in describing spam blogs back to as a minimum, 2003. The “art” developed from many linkblogs that were attempting to manipulate search indexes and others attempting to Google-bomb every word in the dictionary.
It has been estimated that about one in five blogs are spam blogs. These fake blogs waste disk space and bandwidth as well as pollute search engine results, ruining blog search engines and are detrimental to a blogger’s community networking.Google's search engine uses PageRank, which is susceptible to link flooding, especially from highly weighted bloggers.

RSS abuse
Full content RSS feeds make the splog problem worse .As an RSS feed simplifies the coping of content from genuine blogs. Splog RSS feeds pollute RSS search engines, and are reproduced and propagated throughout the Internet.

Defences
A number of splog reporting services have arisen, allowing Internet users to report splog with plans of offering these splog URLs to search engines so that they can be excluded from search results. These services started with Splog Reporter. Some of the main services include:

SplogSpot which actually maintains a large database of Splogs and makes it available to the public via APIs,
A2B blocks web server IP addresses that splog URLs resolve to.
A Feed Copyrighter plugin (for WordPress) allows for the automatic addition of copyright messages to feed, so Splogs can be easily spotted and reported by visitors or through Google search.
TrustRank attempts to automatically find Splogs.
Blogger has implemented a system that can detect Splogs and then force them to take a Captcha 'spell this word' test.

SEO's

2009-07-27T08:07:00.006+10:00

There are "some" valid uses for SEO's, but in reality, this is an area that has degraded into the criminal and unwholesome for the most part.

This varies from the physical attacks that come from human such as “Paid to Read” and “Pay to Click” sources. Looking at correlations from time and location, correlations on browsing patterns etc, one can oft find sources that are likely to be engaging in unacceptable activities that are deployed to promote sites engaged in "click-fraud" and other such actions - the world of the SEO. The more you are able to promote the site - no matter how useless it is, the more you get paid.

If the cost of directly paying or coercing a human to click on a banner is less than the returns of clicking on the banner fraud will occur. Reference sources and referrers sites could be a pornographic site as well with many sites offering access to porn if they solve CAPTCHA’s for instance.

Impression spam comes about as a consequence of HTTP requests for web pages that contain advertisements, but that do not inevitably match up to a user viewing the page. A web crawler or “scraping” program could be used to issue an HTTP re-quest for a web page that happens to contain a banner. Statistically, this variety of request could be distinguished from those requests issued by human users as the banner requests would not correlate to the other images and calls made by the page.

Next there are invalid clicks – these occur due to malicious intent from either “advertiser competitor clicking” or “publisher click inflation”. I see advertiser competitor clicking being the type that is a problem in this instance. For this it would be necessary to analyse click sources against competitor keywords.

Then there is the issue of Tor networks and open Proxies and also those that strip cookies, modify identifying information and change requests. Infected cyber-cafes are also an issue. However, source address location to client market may be used in many cases to determine fraud. Also, over time these are likely to succumb to analysis.

And there are the particular robotic attacks that could be deployed. Analysis of this area involves checking for signs of “bots”. There is a possibility of invalid user-agent string or unlikely fields in the headers of the HTTP requests. The statistical distribution of the user-agent string should have some correlation to the distribution of browsers.

Clickbot networks have a level of predictability as do for-sale/ for-rent botnets.
“Forced browser clicks” are more difficult. This is more likely to require offline detection. The aggregate set of clicks should correlate to the distribution of files on the web server.

Next there are a number of other areas to consider:

Covert_TCP and other covert channel methods,
Rootkits
DNS rebinding attacks
Distributed malware
XSS, Flash with HTTP calls, etc
Splogs

One thing that has occurred in the past has been a strong correlation across the IP addresses in email spam blacklists and click-bots – thus infected hosts may be participating in email spam botnets as well.

Some other considerations in detecting this type of activity would have to include p0wf (Passing Fingerprinting of Web Content Frameworks) and time based correlations.

SPAM

2009-07-27T08:07:00.004+10:00

SPAM is ANY unsolicited commercial email. The issue is commercial. As long as there is a way to make money it is SPAM, anything (such as activist emails for a cause) that does not make money for the parties involved is not by definition SPAM.

So basically, Spamming can be defined as sending unsolicited commercial e-mails (UCE). The more common term for spam is junk mail. Spammers obtain e-mail addresses by harvesting them from Usenet, bots, postings, DNS listings, and/or Web pages.

See also the post on SEOs.

Web Spiders

2009-07-27T07:46:00.002+10:00

I am going to write on two (2) web spiders that I commonly make use of in this post (there really but 2 are related).

First we have Web Scarab or the "WebScarab web auditing tool". This is a project from the OWASP framework and toolsets. It is designed to help ensure that websites are secure by creating a tool to break them (in simple terms). WebScarab is a Java based framework and web proxy designed for analyzing applications that communicate using the HTTP and HTTPS protocols.
The following list of features and functions taken directly from the OWASP site, and details explicitly why WebScarab is such a critical tool to the Web application tester. The any issue with this tool is that it is not for the faint of heart. You will probably find little value in this tool if you cannot already manually test a site. For those who can however this tool takes Web testing to the next level.

Fragments - extracts Scripts and HTML comments from HTML pages as they are seen via the proxy, or other plugins
Proxy - observes traffic between the browser and the web server. The WebScarab proxy is able to observe both HTTP and encrypted HTTPS traffic, by negotiating an SSL connection between WebScarab and the browser instead of simply connecting the browser to the server and allowing an encrypted stream to pass through it. Various proxy plugins have also been developed to allow the operator to control the requests and responses that pass through the proxy.
Manual intercept - allows the user to modify HTTP and HTTPS requests and responses on the fly, before they reach the server or browser.
Beanshell - allows for the execution of arbitrarily complex operations on requests and responses. Anything that can be expressed in Java can be executed.
Reveal hidden fields - sometimes it is easier to modify a hidden field in the page itself, rather than intercepting the request after it has been sent. This plugin simply changes all hidden fields found in HTML pages to text fields, making them visible, and editable.
Bandwidth simulator - allows the user to emulate a slower network, in order to observe how their website would perform when accessed over, say, a modem.
Spider - identifies new URLs on the target site, and fetches them on command.
Manual request - Allows editing and replay of previous requests, or creation of entirely new requests.
SessionID analysis - collects and analyses a number of cookies (and eventually URL-based parameters too) to visually determine the degree of randomness and unpredictability.
Scripted - operators can use BeanShell to write a script to create requests and fetch them from the server. The script can then perform some analysis on the responses, with all the power of the WebScarab Request and Response object model to simplify things.
Parameter fuzzer - performs automated substitution of parameter values that are likely to expose incomplete parameter validation, leading to vulnerabilities like Cross Site Scripting (XSS) and SQL Injection.
Search - allows the user to craft arbitrary BeanShell expressions to identify conversations that should be shown in the list.
Compare - calculates the edit distance between the response bodies of the conversations observed, and a selected baseline conversation. The edit distance is "the number of edits required to transform one document into another". For performance reasons, edits are calculated using word tokens, rather than byte by byte.
SOAP - There is a plugin that parses WSDL, and presents the various functions and the required parameters, allowing them to be edited before being sent to the server.
Extensions - automates checks for files that were mistakenly left in web server's root directory (e.g. .bak, ~, etc). Checks are performed for both, files and directories (e.g. /app/login.jsp will be checked for /app/login.jsp.bak, /app/login.jsp~, /app.zip, /app.tar.gz, etc). Extensions for files and directories can be edited by user.
XSS/CRLF - passive analysis plugin that searches for user-controlled data in HTTP response headers and body to identify potential CRLF injection (HTTP response splitting) and reflected cross-site scripting (XSS) vulnerabilities.

The spider function of Web Scarab allows a tester to download an image of the site for offline tests.

On the other hand, Black Widow and Brown Recluse (each by SoftByte Labs) are dedicated spiders.
Black widow provides a searchable spider with the ability to save page and site structure, it also has a RegEx (regular Expression) based search function.
Brown Recluse is a programmable spider. It can be configured to search sites for selected strings or by server type.

Workshop 6 Enjoying the Ride: Web framework alternatives, scalability and flexibility

2009-07-26T19:26:00.004+10:00

To Do:

Developers may continue to build upon work with the OTBS using the topic reading to help with user registration and advanced login features from Hartl et al (2008).

o generate a controller and an action by adding a method(s) to a controller;
o create a view template for each action and to link to actions from views;
o use AJAX to improve the user experience;

Share your success by posting progress comments and links etc to the Developers sub-forum site that has been set up for the Red team.

As either a developer or as an IT manager, how can aspects of social networking be applied to the OTBS?

Here I have the same issues as in exercise 5.

You can add social networking to the OTBS, but WHY? It seems like adding a monitor to the fridge. A good idea at the time maybe?

Where I fail is in the creation of the user experience. I still like command lines, you may have top learn how it functions, but when you do it is faster. Adding to user experiences is not my forte.

What I have however is a post on code reviews - something I believe (biased) is more important.

Workshop 7 End of the Line: production site migration and maintenance

2009-07-26T19:25:00.006+10:00

To Do:

Developers conclude their work with the OTBS and look at the options for deployment of the site. Examine the various platforms/software tools used for deployment such as UNIX environment suggested in the Discussion Notes, Mongrel or Mongrel cluster, Nginx, Subversion or Capistrano (during development stage), JRuby in the Java environment.

Which way?

The choice is up to you as this workshop present just one option and you may like to use another, such as deploying the OTBS in a .NET or J2EE environment

Can you get the OTBS Running in production mode as a minimal production server?

Share your success by posting progress comments and links etc to the Developers sub-forum site that has been set up for the Red team.

Focus Question
As either a developer or as an IT manager, what are the options available when deploying and maintaining the Ruby on Rails application online?

For me, there are two (2) options (a third if you include Solaris, but I have not tried Ruby on Solaris yet): This is as I use the following IDEs

Eclipse (on Linux)
Visual Studio 2008 (Windows)

As you can see from the figure below, I have installed Ruby on Steele and Iron Ruby into Visual Studio 2008.
On Linux add to this:

Sunversion, &
TortoiseSVN

These are source code repositories. These are similar in function to Microsoft Windows with Visual SourceSafe, but they work.

In addition to this, I have to mention something just as critical as the code SVN, a CASE tool that can also maintain the database. To this end I have included a post on Case tools and in particular the ones I use:

XCase
DBAnalyser

The post can be found here.

Workshop 8 Ruby on Rails Workshops Report and Evaluation

2009-07-26T19:25:00.005+10:00

Upon the completion of this workshop, developers or managers should be able to:

Identify and evaluate the Ruby on Rails workshop series
Think critically and analytically about what you knew before and after the experiences
Share and post your Report and Evaluation with peers via the subject forum.

Evaluation and Report

Please answer each question in this evaluation section. In your answer, please consider content/topics presented and the technologies and teaching strategies used during the Ruby on Rails Workshops. Results will be collated and used to modify the workshop series.
This form is just a format guide to you evaluation and report. Thank you for your time to complete workshop 8.

1. List what you consider to be the three strengths of Ruby on Rails workshop series

Quick and simple
Graphically rich
Easy to read and decode

2. List what you consider to be the three weaknesses of Ruby on Rails workshop series:

Slow
Slow
Slow

In addition to this, it is easy to reverse engineer - a good point for me, but not to all developers. With the Steele integration platform for VisualStudio it is possible to compile Ruby and it does go faster this way, but not a good deal.

There is also little focus on security or references on how to secure Ruby to an acceptable level.

3. List what aspects of Ruby on Rails workshop series that you found to be most difficult.
Graphics. The text ideas I can do well, but I have never been a graphically focused programmer, hence why I have ended up reversing malware and designing algorithms.

4. List what improvements could be made to the Ruby on Rails workshop series:
From the POV of Ruby, I have no complaints.

Free response and reflective questions:

5. Reflect on your experiences with the other Web framework used in this subject: Was it effective? How can it be improved? Should other Web frameworks be used as well or instead of Ruby on Rails?
PHP has a number of frameworks. Like it or not, these are common. I do not see them being replaced any time soon.

6. Did the Developer’s or IT managers Team that you joined after workshop 4 have a preference towards using other tools to facilitate collaboration? Comment on the differences between these use of the sub-forum or Interact wiki tools from your experiences in this subject.
Not really, we sort of all went off on our own, seem much like a standard development project - herding cats.

7. Further comments to add?
I will stick to C and C++.

Workshop 5 Admiring the scenery Forms, AJAX screen layout and mobile interfaces

2009-07-26T19:20:00.003+10:00

Like the majority of the class, I started this session and workshop as a developer in place of choosing management. I thought of changing, but no.

I have at this stage to report my findings. What I do have to state is that I would prefer not to leave the processes to the web server so much. I (being a dinosaur) like to set triggers in the database itself.

I also find that there is too much trust in the Ruby world as to what people receive.

NEVER trust the data from the client.

There are ways to lock some of this down, but they are not defined nor taught. This is an area I have concerns with. I mentioned web scarab in a previous post. This tool can allow a malicious user (or a tester for that matter) to alter the form fields and data that they receive.

I find that there is little effort to teach good coding practice where errors are set and logged with alerts when this occurs. Just as the user does not have an option to select "barn" under the option "What type of building is at your address?" on the example exercise page does not mean that the user can not attempt to send this.

If this occurs, what happens. Does the database add it, is there an error, does it crash. All of these are the questions that we need to start to consider early.

Exercise 25: M-commerce and the e-wallet: Innovation and mobile devices

2009-07-26T10:08:00.005+10:00

Explore some of the problems associated with mobile technology or their suppliers.

What is meant by a location based service?
Visit an airline Web site and search for information on WAP or SMS access to booking airline services. Do the same for WAP or SMS services in banking. How do both industries compare?
Lucent Technologies designs and delivers the systems, services and software that drive next-generation communications networks at: http://www.lucent.com
Visit the W3C website and find the status of the VoiceXML project. When do you think it will affect business on the Web and what will its impact be?
Investigate CDMA, GSM or other network technologies for mobile phones and circuit-switched and packet-switched data capabilities.
According to Nokia:

“The Nokia One Mobile Connectivity Service provides easy and secure access to email, calendar, directory and more from a mobile phone, PDA, PC or fixed-line phone - take your corporate applications mobile.”
Why is a company like Nokia – http://www.nokia.com – described as having end-to-end expertise?

According to Takeshi Natsuno, media director of Gateway Business Department, NTT Mobile Communications Network, Inc (NTT DoCoMo) of Japan:

“The mobile phone will be an electronic wallet.”

Parts 3, 4 and 6 are at present solutions without a niche to fill. VoIP may have a place (for Lucent) but the overall push to see what can be integrated is a typical technology centered approach to "if you build it they will come".

The truth is that most technology fails.

The reality is that we will move towards integrated mobile platforms, not the phone. CPU time gets cheaper, memory gets cheaper, soon the computer will replace the phone (in my case it to a certain extent has already).

The great part of all this is that data will (and is becoming) be king. Who cares where traffic if going to or from with a data based session. This is an issue for the Telcos (eg Telstra) as this also means no more long distance call charges, but they will gradually move to a new model.

Exercise 26

2009-07-26T10:08:00.004+10:00

To take the last forum quote,

Is it in Topic 12?

I will check. There may be just 25 during the last edit.

-Ken

It would appear that there is no exercise 26.

Exercise 24: Virtual business worlds and cyberagents

2009-07-26T10:04:00.001+10:00

Search the Web for a site that uses a cyber character or cyber agent to host a business site. (If you create a successful cyber agent, you may be able to get large companies to use it to sell their products online.)

Describe what software agents are.
Differentiate the various types of software agents.
Describe how techniques such as artificial intelligence and statistical techniques are used in software agents.
List popular software agents currently in use in the commercial world.
Identify various activities in e-commerce where software agents are currently in use.

I have decided to approach these final couple exercises a little differently. I have added a little from the legal side of the issue and also (as others have covered this well directly), have approached the question a little askew.

Dal Pont recognizes three possible categories of agents:

those that can create legal relations on behalf of a principal with a third party;
those that can affect legal relations on behalf of a principal with a third party; and
a person who has authority to act on behalf of a principal.

At the end of the day, an agent is a party who acts on the behalf of a principle. A software agent is simply but a set of programmed tasks that are set to be enacted by the agent.

Electronic agency issues
The inclusion of electronic agents makes the traditional requirement for a "meeting of minds" more difficult to prove. With many smaller vendors, hosting and creating their own e-commerce enabled web site requires the interaction of a third party. Often, this involves the use of an external service provider, which offloads the Internet shopping trolley function. In this way, smaller vendors can create an e-commerce enabled site quickly and simply.

The issue, which arises in this instance, is in determining the contracting parties. Many small vendors provide little more than billboards style advertising through their web site. The complex task of maintaining the databases, transaction processing, and the shopping cart function becomes simplified when outsourced to another provider. In some instances, a redirection takes the customer to a completely new site or domain.

In such cases, it may be necessary to investigate whether a contractual arrangement has resulted between the client browsing a web site and the transaction agent or if indeed the transaction facilitator is a contractual agent for the Web store vendor . Agency has become a specialized area of contract law in itself. As such it will not be covered in any depth in this paper, though it is an area that does require due consideration and may influence the process of offer and acceptance.

Basically, an agent can be created for any purpose that can be completed using a set of logical (including fuzzy logic) processes. In a web 2.0 context, this would generally relate to database processes (but not necessarily). A database trigger or SQL stored procedure are a couple ways of achieving this process.

I have defined triggers in more detail in the linked post.

References

G. E. Dal Pont, (2001) “Law of Agency”, Butterworths

Exercise 23: Searching mechanisms

2009-07-26T10:03:00.003+10:00

How do search engines such as Alta Vista differ from information directories?
What is a spider? What does it do?
Describe a search situation where the requirement for recall is high?
What is a meta-search engine? Provide some examples.
What is spamming?
How can you get your site listed at major search sites; and how could you improve your site ranking?

See the links.

As for the others...

1. Information directories are a specialised repository of selected knowledge. The ACM library, IEEE paper archive and similar sites act in such a manner. Alta Vista, Google, Bing etc are search indexing services. These are databases of other sites and material that can be used for online seraches across the entire (or a large segment) of the Internet.

3. High recall comes into play when the ability to find the information is critical. Such a situation occurs when using specialised search tools such as medical databases or legal precedence searchers. One such example is Westlaw.

4. Meta-search engines are used to search across multiple Internet search providers. For the most part they are junk filled wastes of time that offer little and consume time. The underground engine, Alstalavista (not to be confused with the more commonly known Alta Vista) is one such example.

For security research, it can become necessary to search for "hacker" tools and this site aides in this process.

As a reference to both web site optimization and SPAM, I have a post on Splogging that I wrote in the past on one of my other blogs. Follow the link to read it.

Exercise 22

2009-07-26T10:01:00.001+10:00

Use the coordination theory framework to describe the contribution of ERP software to organisational goals such as efficiency and flexibility.
Differentiate between software systems such as Customer Relationship Management (CRM) software, Business-to-Business e-commerce programs and Supply-Chain Management (SCM) software.
What are the limitations of the EDI platform? How does a web-based platform for inter-enterprise communication rectify these limitations?
Describe the CRM Life Cycle and the different segments of CRM software.

I will start by pointing to a post I have made in the past - Principles of Database, Datawarehouse and Repository Development.

This has a different focus to most of the group, but it is where I am interested and hence put my time.

The text of the link follows:

Introduction
As the main repository of an organisation’s historical data, the data warehouse is evolving into the memory of a corporation[1]. Through storage of a wide variety of data sources in an integrated format, the data warehouse is becoming both the storeroom of past events, and the central predictive engine.

The data warehouse contains the unrefined substance that when fed through a decision support system can provide management with up-to-date analytics and corporate predictors. The data analyst can use this technology to perform complex queries and analysis of information without adversely impacting operational systems.

Through both the rise in computational speed and power and the growth of data storage, data warehouses have pressed other technologies into greater levels of development[2]. Technologies such as data mining have grown symbiotically with data warehouses due to the benefits that they can provide. To understand data warehousing it is necessary to have knowledge of both data warehouse technologies and the associated analytical analysis methods used to access, report and present on the data.

The data warehouse architecture illustrates the entire organisational process from a variety of points of view[3]. These include the data, processes and infrastructure of the organisation and can mirror the structure, function and interrelationships of each constituent element of the organisation.

Data warehousing and Data analysis
The infrastructure or technology viewpoint reflects the choice of hardware and software products as they are implemented by the distinct components which derive the overall system. The data perspective characteristically epitomizes the source and target data formation and can assist the members of the organisation to comprehend the data assets and functional relationships which make up the organisation’s operations. The process viewpoint is principally focused on the communication of the progression of data from the originating source database through to the procedure to load the data into data warehouse and finally to analyse and extract data from the warehouse.

To be able to explore the effect of the rise of data warehouses on business and society, one first needs to characterise what the concept encompasses. William H. Inmon[4], known as one of the fathers of data warehousing, has stated that data warehouses are required to be subject orientated. In this, the data and thus database is organised in a manner such that all data relations relating to the same event or object are associated correspondingly in a manner that is concurrently time variant, non-volatile and integrated.

Whereas operational systems are necessarily optimised for ease of use and the rapidity of response, the data warehouse is optimised for reporting and analysis. Online transaction processing (OLTP), crucial to the operational system, is of less importance to the data warehouse. Rather, on line analytical processing (OLAP) and the necessity to access unusual data patterns results in heavily denormalised or dimension based models that may not be required to achieve acceptable query response times[5].

By time variant, is meant that changes to the objects and tables within the database are tracked and evidenced. This process allows for the statistical analysis of the data over time to produce reports on time variant trends. Heteroscadastic (including ARIMA, GARCH and ARCH) time series analysis of data is one of the newer avenues of research.

A data warehouse requires that once data is committed to the database, it is stored as read only. In this it is used for future reporting and any point in time data becomes an individual snapshot of the database over time. This process allows both for historical analysis and also future predictions.

To be effective, and data warehouse needs to contain data from as many if not all of an organisations operational applications and individual databases[6]. Further, to be of any effective use, the data in the warehouse must be contained in a consistent manner. A failure to either constrain data consistently or to provide an adequate sample of the organisation’s data leads to the GIGO issue[7]. That is, garbage in, garbage out.

Research into data warehousing has expanded into what has been termed the Corporate Information Factory (or "CIF"). A CIF[8] is an organisational data structure which encompasses ERP, eCommerce, customer relationship management (CRM) and many other formerly separate reporting structures. In some cases, a CIF has been known to encompass data marts, exploration warehouses, ODS, both nearline and secondary storage, and project warehouses.

However, the volumes of data, expense and lack of enterprise support leave CIF implementations is an idea for the future. There are still a number of difficulties associated with data warehousing which make their implementation less widespread then maybe expected in the future. Of particular concern, the process of extracting, cleaning and processing data is both time-consuming and difficult. The failure to implement and adhere to corporate wide naming standards amongst many organisations as only exasperated this problem.

Widespread incompatibility between many database products has slowed down the broadening of data warehousing across organisations. Technologies such as OLAP have aided in the development of Cross-application data warehouses[9], but issues of table structure, normalisation and the type of data stored within individual databases remains an issue.
Another issue that has hampered the implementation of data warehousing is security. In a world that is increasingly becoming reliant on the Internet and the Web, security could develop into a serious issue. With links into the data warehouse from the Internet, an organisations key informational assets are at risk from both discovery and compromise.

There are a number of clear advantages to the adoption of data warehouse technologies[10]. It is easy to see that organisations which adopt these new technologies successfully will gain a clear competitive advantage. These technologies enhance end user access to data and reports in a manner that allows for greater creativity and more informed evaluations.

The ability to create trend reports and use statistical methods to accurately forecast probabilistic events based on the past occurrences and experience provides for more focused corporate activity[11]. For instance, marketing information from a particular sales push can be compared across different regions to evaluate the impact of differing advertising initiatives.
Data warehousing technology can also significantly increase the effectiveness of several commercial business applications. In particular, customer relationship management (CRM) benefits greatly from this technology. The ability to both gain an overall view and to be able to drill down to specific areas and individuals provides a significant advantage to many organisations. CRM has been one of the principal applications to make early use of data warehousing.

Data Mining
Data mining is a relatively new development. The use of statistical methods to routinely investigate large capacities of data for patterns by using techniques including classification, decision trees[12], association rule mining, and clustering, has resulted in a new field of computational mathematics[13]. Data mining is a complex subject in itself and has associations with numerous core disciplines together with computer science and appends significance to influential computational techniques from statistics, information retrieval, machine learning and pattern recognition[14].

Too many people, the key issue introduced through the use of data mining is not commercial or technological in nature. It is a social issue. The protection of individual privacy is a concern that has increased exponentially with the rise in data mining. Data mining increases the possibility of an individual’s privacy being violated in some manner.

The processes used to analyse ordinary commercial transactions may be used to compile significant quantities of information about individuals from their purchasing behaviour and lifestyle preferences[15]. In particular, where data is compiled across multiple organisations that need to protect the privacy of individuals identity is compounded.

There are five primary stages to data mining. Initially it is essential that the data is loaded into the data warehouse system. This phase includes the extraction of data from the source databases and any necessary transformations required to reformat the data or cleanse the data such that it is maintained consistently.

The next phase involves storing and managing the data. In this, the data needs to be normalised and formatted such that it fulfils the requirements necessary to maintaining a multidimensional database system. This data must be accessible to the organisation’s business analysts. The process of providing access to the data is the third phase of data mining. No data warehouse project may be considered successful if the business is unable to access the data.
The final two phases of data mining call for analysis and presentation. Using methods such as OLAP, ROLAP and MOLAP[16] to access the data, the business analyst will load the required information from the data base into specialised application software[17]. These products are then able to present the data contained in the data warehouse in a more usable format such as the graph or table.

The Types of Analysis
The data maintained in a data warehouse is of little use if it cannot be accessed and analysed. As a consequence, a number of analysis techniques have developed. Although not exclusive, the following provides a brief summary of the analysis techniques available for use against data warehouses.

Decision tree methodology has been around for a long time as a probabilistic tool. This technique uses tree shaped structures to represent a variety of decisions and possible outcomes. This process generates rules for the classification of data sets. There are a number of specific decision tree methods including Classification and Regression Trees (CART) and Chi Square Automatic Interaction Detection (CHAID) which have been derived from probability theory[18]. These methods provide a set of rules which may be applied to the data set in order to predict or forecast a given outcome from the data.

The CART methodology segments a dataset by creating 2-way splits[19], whereas the CHAID method segments using chi square tests to create multi-directional splits. CART methodologies are more common than CHAID methods as they typically require less data preparation. However, CHAID methods can provide a greater level of statistical precision[20].

Another method is the nearest neighbour method. This technique classifies each record in the dataset using an arrangement of the classes of the “k”[21] records which are the most comparable to it in a historical dataset. This technique is also known as the k-nearest neighbour technique[22].

Rule induction is common in organisations with a strong programming all logic background or focus. This technique uses a variety of predetermined statistical tests to extract data using “if-then-else” rules[23]. Analysis programs such as those provided by SAS and the open source product “R” make extensive use of this method.

A further common method is “data visualisation”. In this involves the generation of crafts and reports which allow for the visual interpretation of compound associations in multidimensional data by the analyst. Graphics tools are used to illustrate relationships between the data in a manner that provides more straightforward reporting than many of the other methods.
With the advances in both computational power and the development of new mathematical techniques, a couple of advanced analysis methodologies have developed including artificial neural networks and genetic algorithms. Artificial neural networks used non linear predictive models which “learn” using training algorithms to provide intricate statistical reports on the data. These techniques are so named as they resemble biological neural networks in their structure.

Genetic algorithms which use optimisation techniques and algorithmic evolution are also developing. These processes use a combination of “genetic combination”, mutation and probabilistic methods to simulate natural selection. These methods commonly integrate stochastic techniques, such as Monte Carlo simulations to provide estimates and forecasts from the data contained in the data warehouse[24].

Conclusion
Due to the exponential growth in the volume of data and the continuing development of new techniques for data analysis, organisations are continuing to bring about changes in the methods they use to both store data and analyse it. Data warehouse architectures which are now primarily used to express the overall configuration of a Business Intelligence system have integrated decision support systems (DSS), management information systems (MIS), into their fold giving businesses access to more information and predictive capability than ever before.
This rapid increase in data presents many problems for organisations, but at the same time provides opportunities for those who know how to use these new technologies and techniques. Those businesses and organisations which most effectively make use of this new technology are likely to gain significant competitive advantages. As a result, the implementation of data warehousing techniques and technologies is only likely to continue.

Coupled with the advances in analytical technologies such as neural network analysis and genetic algorithms, organisations now have greater access to the data than ever before. This increasing level of access has become the new organisational paradigm.

Bibliography

Agresti, A. (1990), “Categorical Data Analysis”, New York: John Wiley & Sons.
Amado, Carlos Armando (Miami, FL, US) (1997) “Method and apparatus for applying if-then-else rules to data sets in a relational data base and generating from the results of application of said rules a database of diagnostics linked to said data sets to aid executive analysis of financial data” US Patent Office – Application No. 400355 - G06F 015/18
Berson, A & Smith, A (1997) “Data Warehousing, Data Mining and OLAP” McGraw Hill USA
Beynon-Davis, P., 2004. Database systems, 3rd edn, Palgrave McMillan.
Codd, E & Codd, S (1993) “Providing OLAP to User-Analysts: An IT Mandate”, Comshare
Collett, D. (1991), “Modelling Binary Data”, London: Chapman & Hall.
Date, C.J., 2004. An introduction to database systems, 8th edn, Addison Wesley.\
Fair Issac (2003) “A Discussion of Data Analysis, Prediction and Decision Techniques”
Fair Isaac White Paper, May 2003; http:// http://www.fairisaac.com/
Frank M. (1994) “A Drill Down Analysis of Multi-dimensional Databases”, DBMS, July.
Hoffer, J., Prescott, M., McFadden, F., 2007. Modern database management, 8th edn, Prentice Hall.
Inmon, W. H. (1995) “What is a Data Warehouse” Prisim Solutions Inc, http://www.cait.wustl.edu/cait/papers/prism/vol1_no1?
Inmon, W. H. (1996) “What is a Data Mart”, Informatiques Magazine. Avril
Inmon, W.H.(1996-2) “User Reaction to the Data Warehouse.” DMR (December 1996).
Inmon, W. H. (2003) “Building the Data Warehouse” Wiley Computer Publishing, USA
Keith, Steven; Kaser, Owen & Lemire, Daniel (2005) “Analyzing Large Collections of Electronic Text Using OLAP”, UNBSJ CSAS, TR-05-001, 2005.
Kimball, Ralph & Ross, Margy (2002) “The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling” 2nd Ed. Wiley Computer Publishing, USA
Kimball R. (1997) “A Dimensional Modelling Manifesto” DBMS Online, http://www.dbmsmag.com/9708d15.html
Kroenke, D., (2003) “Database processing: Fundamentals, design and implementation”, 10th edn, Prentice Hall.
Lehn, R., Lambert, V. & Nachouki, M.-P. (1997) "Data warehousing tool's architecture: from multidimensional analysis to data mining," dexa, p. 636, 8th International Workshop on Database and Expert Systems Applications (DEXA '97).
Ma, Yao (1998) “Data Warehousing, OLAP and Data Mining; An Integrated Strategy for use at FAA”, M.Eng Thesis, MIT 2nd edn, McGraw-Hill.
Hegland, M (2001) “Data mining techniques” Acta Numerica (2001), Volume 10: Pp 313-355 Cambridge University Press
Mento, B & Rapple, B (2003) “Data Mining and Data Warehousing” Association of Research Libraries, US
Nguyen, Tho Manh & Tjoa, A Min (2006) “Zero-Latency Data Warehousing (ZLDWH): the State-of-the-art and experimental implementation approaches” Institute of Software Technology and Interactive Systems, Vienna University of Technology Favoritenstraße 9-11/188-3 (2. Stock), 1040 Vienna, Austria
Pokorny J. (1998) “Conceptual Modelling in OLAP”, Proceeding of ECIS’98, Aix-en-Provence, Pp 273-288.
Pratt, P. & Adamski, J., (2005). “The concepts of database management”, 5th edn, ITP.
Pulleyblank, W. R. (2002) “Mathematical sciences in the nineties” Systems Journal, IBM Journal of Research and Development, “Mathematical Sciences at 40” IBM, Vol. 47, No. 1, 2003
Service, R & Maddux, H, (1999) “Building competitive advantage through IS: the organizational Information Quotient”, Journal of Information Science25(1) Pp 51-65.
Sullivan, D. (2001) “Document Warehousing and Text Mining: Techniques for Improving Business Operations”, Marketing, and Sales, USA
Thalhammer, T; Schrefl, M. & Mohania, M (2001) “Active Data Warehouses: Complementing OLAP with Analysis Rules”, Data & Knowledge Engineering, Elsevier Science Ltd., Vol. 39(3), Pp. 241–269.
Vassiliadis, P; Quix, C; Vassiliou, Y & Jarke, M (2001) “DATA WAREHOUSE PROCESS MANAGEMENT” Informatik, De
Whitehouse, Peter R. (2006) “Case Studies in Database Design and Implementation” UQ Australia

Exercise 21: Shopping cart specifications

2009-07-26T09:59:00.002+10:00

Develop the class diagram for the following shopping cart specifications:

A shoppingCart object is associated with only one creditCard and customer and to items in itemToBuy object. Persistent customer information such as name, billing address, delivery address, e-mail address and credit rating is stored in the customer object. The credit card object is associated with a frequentShopper discount object, if the credit rating for the customer is good. The customer can make or cancel orders as well as add and delete items to the shopping cart product. The credit card object contains the secure method for checking that the charge is authentic.

Here I am using Altova's UModel again to create a class diagram (as was noted in exercise 20).

Quick and nasty, but I am not really designing this to last ...

# What is difference in load balancing with traditional and transactional MOM, RPC and conversations?

2009-07-23T21:21:00.001+10:00

I am going to take the easy way out on this one as a paper I read sums this up fairly succinctly.

This paper was:

Qiyang Chen, James Yao, and Rubin Xin (2006) "Middleware Components for E-commerce Infrastructure: An Analytical Review", Issues in Informing Science and Information Technology Volume 3, 2006.

This paper has a fairly quick and simple summary of the various technologies.

For instance, to take two (2) patent blockquotes from this (MOM and RPC) we have:

Remote Procedure Calls (RPC) permits a client program to call procedures located on a remote server program. Remote procedure calls is not isolated as distinct middleware level and is entrenched into the application with calls embedded into the client portion of the client/server application program. Stubs are developed for both the client and the server to call up synchronously when the client makes a call to the server. The intricacies of distributed processing are reduced by remote procedure calls by maintaining the semantics of a remote call no matter the client and server are located on the same system or not. The synchronous nature of the remote procedure calls makes it most appropriate for smaller applications where all communications are one-to-one and not asynchronous (Slater 2002).

Which we can make a comparision against MOM from the above definition of RPC with that below:

Message Oriented Middleware (MOM) or enterprise message technology (EMT), provides asynchronous message delivery. The messages are lined up, just as objects, permitting the application that sends messages, to carry out other tasks without getting blocked till it receives the response. Generally located at a higher level than that of remote procedure calls, MOM assembly provides more than simply passing information. MOM also offers provisions for translating data, security, broadcasting data to multiple program, error recovery, and prioritization of messages and requests.
MOM enhances flexibility by allowing applications to switch messages without the requirement of knowing on which platform or processor the other application located. MOM facilitates communications across a range of messaging systems, such as request-response, prolonged conversation, application queues, publishing and subscribing, and broadcasting. Messages are processed asynchronously with appropriate priority levels. Examples of messaging middleware are IBM's MQSeries, BEA’s Message Q, and Microsoft’s MSMQ.

Why is a two-phase commit protocol better than a one-phase atomic commit protocol?

2009-07-23T21:20:00.001+10:00

Basically, it is a fairly simple answer that leads to a complex means of integrating and creating this solution.

This is integrity.

Atomic solutions concern ones that work and are accepted in one instantiation, or are reversed in full. A two-phase commit protocol ensures (or at least should ensure) that there is a roll back process (such as transaction logs in SQL systems). Where the complete transaction has not been fulfilled, there is a record of the transaction state and ti can be rolled back to the state of the system as if it did not occur. This is a process that is far more difficult (and at time impractice) when using a one-phase transaction system.

Event

2009-07-23T15:41:00.004+10:00

In, Ferg (2006) we see "a stream of data items called events (Yourdon and Constantine's "transactions")". An event is any occurrence that can impact the program from entering a transaction to moving a mouse or a timer counting down.

If you can get past the fact it is from a Mac system focus, also see:
http://www.mactech.com/articles/mactech/Vol.08/08.07/EventProgramming/index.html

See:
Ferg, Stephen (2006) "Event-Driven Programming: Introduction, Tutorial, History"
http://Tutorial_EventDrivenProgramming.sourceforge.net
(steve@ferg.org)
http://eventdrivenpgm.sourceforge.net/

Waitable timer

2009-07-23T15:41:00.002+10:00

As defined by Microsoft;

"A waitable timer object is a synchronization object whose state is set to signaled when the specified due time arrives. There are two types of waitable timers that can be created: manual-reset and synchronization. A timer of either type can also be a periodic timer."

Mutex (mutual exclusion)

2009-07-23T15:39:00.003+10:00

A Mutual Exclusion or "mutex" algorithm is one that is designed to forbid (where possible) the simultaneous use of a common resource in concurrent programming. These resources can include global variables and shared memory.

This is achieved through the use of critical sections (although the critical section itself does not forbid deadlocks or provide for mutual exclusion). These are sections of code in which the process or thread requires shared access to common resources.

Semaphores

2009-07-23T15:39:00.002+10:00

A semaphore is a protected variable or abstract data type that was defined by Turing prize winner, Edsger Dijkstra.

Semaphores have become the classic method for restricting access to shared resources such as shared memory in multiprogramming environments.

A counting semaphore is a counter for a set of available resources, rather than a locked/unlocked flag of a single resource.

Semaphores are the classic solution to preventing race conditions in the dining philosophers problem but they do not stop resource deadlocks from occurring.

Semaphores can only be accessed through the operations listed below:

P(Semaphore s) // Acquire Resource
V(Semaphore s) // Release Resource
Init(Semaphore s, Integer v)

Where a semaphore is tagged as being atomic, it should not be interrupted. This is, the process should complete or not start. It should not be allowed to finish in the middle of an instruction set.

See:
http://www.doc.ic.ac.uk/~jnm/concurrency/classes/Diners/Diners.html

Deadlock

2009-07-23T15:38:00.002+10:00

A deadlock is the situation where two or more processes each seek to access a resource at the same time. In this instance, the resource is locked, but neither process is set to release the resource.

This can occur as the processes have ended up in a resource loop where each of them is locked until a required resource is released, forming a circular chain.

Locks

2009-07-23T15:36:00.002+10:00

Simply put, a lock is a synchronization mechanism that can be used in order to place and control and thus limit that amount of access that is available to a resource in a multi-threaded environment.

Hence, a locks enforces the concurrency of the system and maintains its control policies.

Thread Synchronisation

2009-07-23T08:30:00.002+10:00

The entire notion and issue with threads comes from working within a multitasking environment. Windows systems time-slice, they are not truly running parallel processes in separate environments (as some old iron used to do). As processes are swapped back and forth at blindingly fast rates, the system is required to manage these. That is, it is crucial that the operating system coordinates the execution of multiple threads that may lie over one or more processes that are running on the system at any given time.

The management of these tasks and the allocation of time slices (small fragments of CPU time and other resources) make the operation of the OS seem (at least conceptually) to run as a concurrent process. In newer multicore systems and SMP (symmetric multi-processor) systems, processes can truly run concurrently as the individual cores or processors act without direct reference to the others. This of course requires that the threads created by the programmer are carefully managed.

To do this, systems provide a scheduler. The scheduler can pre-empt a thread at any time. This can be a major problem as the scheduler can cause a to access data without regard to serialisation leading to race conditions and other security concerns. An example would be a system that returned data before the authorisation process had completed – hence allowing unauthorised access to data.“Serially reusable resources” (SRR’s) are prone to such problems. If SRR’s are not used “one thread at a time”, data can become corrupted, system deadlocks can occur, or worse, race conditions can lead to security failures.

As such, the coordination of coordinate threads is critical to not only the performance of the process, but also the security of the application. Whenever any thread is required to wait for the completion of another thread, a failure to control the order that the threads are processed could create errors. For instance, consider the following algorithmic operations:

(A+B)*C
A+B*C

In case (1), we obtain a result that is rarely equal to that in case (2). Just as in math order can be important, so is order in the execution of threads.

A process, known as “arbitration” or as it is alternatively termed, “mutual exclusion” is used by subsystems in order to make certain that an SRR can only be accessed by a single thread at a time and that only the correct thread is accessing the system. This thread coordination process is termed as thread synchronization.

But what is a Thread?
If we are going to discuss thread syncronisation, we really need to look into what a thread actually is.

Basically, a thread is a fork of an execution path. This is, when a computer program needs to run multiple (two or more) processes simultaneously (this is concurrently) , it creates a fork. Multiple threads are capable of existing within the same process. These can even share resources (including disk, memory, etc). This differs from separate processes which do not generally share these resources.

ACID

2009-07-21T10:51:00.002+10:00

The goal of maintaining database transaction integrity is to ensure that no unauthorized changes occur either through user interaction or system error. In general process following well accepted properties is called the ACID principle.

The ACID principle stands for:

Atomic,
Consistent,
Isolated, and
Durable.

This means that the individual transactions cannot be subdivided, hence atomic. A process must be included in its entirety or not at all.

Next a transaction needs to be consistent. This means that any database constraints must be true. Before the transaction must also be true post the transaction.

Next transactions should be isolated. This means that changes to the database are not revealed to users until the transaction is committed to the database. And finally transactions need to be durable.

Durable transactions means the change has to be permanent. Once a transaction is committed no subsequent failure of the database will end up in reversing the effect of the transaction. This is important in case of failures where transactions may be lost.

Browser security

2009-07-21T10:33:00.003+10:00

FireFox 3.x has incorporated a series of tools designed to aid in providing some level of built-in phishing and malware protection. When a site has been reported to be a forgery of a valid site (phishing) or is known to contain malicious applications (malware), FireFox has been configured to alert the user. This works well for sophisticated users but can be problematic with less-experienced people. This protection can be used to help restrict known “bad” sites from installing malicious code on a user’s system, extending the phishing protection provided in FireFox 2.x.

This feature worked well previously when sites were reported. Blacklists are automatically downloaded and updated to the browser at what approximates to a 30-minute interval. This does require that the phishing and malware protection features of the browser are enabled. Unfortunately, the versatility of FireFox and its lack of integration directly into Microsoft Active Directory increase the likelihood that a local user could bypass these features. The protocol used to update the blacklist is publicly available.

Mozilla has co-ordinated with Google to provide this list. The blacklist is managed by StopBadware. The blacklist contains known Spyware, Adware and other malware vendors and sites. StopBadware is sponsored via funding from Google, Lenovo and Sun.

This feature communicates with both Mozilla hosts and a number of partner systems. List updates do not communicate information about sites that are being visited by the user. In the event that the user attempts to browse a website that is included on the blacklist, the browser will verify that the site is still on the blacklist and has not been removed since the last update. Subsequent to this check, this site will be blocked if it remains on the list. This process will send the user’s cookies associated with “google.com” as a part of the request.

Phishing and malware protection features are enabled on the browser by default. They will run unless the system user or administrator disables them. The phishing and malware protection options are located under FireFox’s Security Preferences pane. To select this, the following menu options are available by “Operating System:”

Overall, FireFox’s blacklist functionality operates in an analogous fashion to that of most e-mail servers and existing anti-spam software. It is necessary that a website has previously been reported to the list as a malicious site and tested before it will be blocked. Once this has occurred, FireFox will warn or block others from accessing this site. The difficulty comes with the less advanced integration of the controls into Active Directory. In a large Windows domain environment, management of the features is difficult.

FireFox’s developing utilization of Vista parental controls is useful for the home user. However, this feature is of little use in a network environment with the difficulty of controlling it over a large network of systems.

Third-Party Add-Ons
Browser extensions pose an immense threat to the security of a user’s system. The ease with which a malicious control can be used to subvert an inexperienced or unconcerned user’s browser is only increasing. As such, both IE8 and FireFox 3.x have implemented controls that are designed to manage which extensions can be loaded and run in the browser.

The aspect of FireFox 3.x that demonstrates the clearest advantage over IE8 is in the ability to manage add-on and extension versioning. FireFox is configured to validate and update all extensions to the browser, automatically patching supported extensions whenever the browser is started. Users can also select “Tools -> Add-Ons” and click “Update” to have FireFox check the Web for updates and install these.

The FireCat project for FireFox is designed as a “mindmap collection of the most efficient and useful FireFox extensions oriented application security auditing and assessment.” With the forthcoming version 2.0 of this project, FireCAT will incorporate an advanced “management of plug-ins, instant download from security-database, ability to add new extension, extension version checker, FireFox 3.X compatible extensions.”

For information on FireCat, see:

http://toolbar.netcraft.com/
http://www.security-database.com/toolswatch/IMG/png/FireCAT1.4.png
http://www.security-database.com/toolswatch for the FireCat project.

No-Script
The security controls and extensions developed both commercially and as open-source releases for FireFox are multitudinous. Extensions such as No-Script (see the image below) and Firebug can be used to turn the browser into a Malware analysis platform. These extensions (and others of the same class) allow a sophisticated user to finely control the actions of scripts and active code in their browsers.

Identification and Authentication

2009-07-21T10:23:00.001+10:00

Identification for the purposes of these standards relates to the way an individual is identified to the system. Logon security is the method commonly used to identify the user accessing the system.

The purpose of authentication is to verify that the person trying to access a system are who they say they are. Authentication is typically verified by the user supplying a password. Other methods of Authentication may be defined as well.

So of these other methods include:

Certificates and smartcards.
OTP (one time pads).
Secure Tokens (e.g. Securid by RSA).
Biometrics (e.g. fingerprint scanners).
Site trusts (this could be something as easy to spoof as an IP address - see rhost)

Access Control
Access controls are provided by software protection measures or procedures to control access to system applications and information according to an organisation’s specified rules.

Access controls are prevention measures designed to:

prevent and minimise threats;
ensure only authorised users have access to systems and information; and
ensure information is protected according to the classification levels.

Passwords are problematic as they are rarely chosen well. This allows tools such as the ones listed below to attempt to crack them.

Brutus: A network brute-force authentication cracker
This Windows-only cracker bangs against network services of remote systems trying to guess passwords by using a dictionary and permutations thereof. It supports HTTP, POP3, FTP, SMB, TELNET, IMAP, NTP, and more. No source code is available. UNIX users should take a look at THC-Hydra.
Available from http://www.hoobie.net/brutus/

THC-Hydra: Parallized network authentication cracker
This tool allows for rapid dictionary attacks against network login systems, including FTP, POP3, IMAP, Netbios, Telnet, HTTP Auth, LDAP NNTP, VNC, ICQ, Socks5, PCNFS, and more. It includes SSL support and is apparently now part of Nessus.
Available from http://www.thc.org/releases.php

Exercise 16, Part 3

2009-07-21T10:10:00.005+10:00

Finally for Exercise 16, Part 3.

Get the latest PGP software from http://web.mit.edu/network/pgp.html; install it on two machines and encrypt a message on one machine and decrypt it on the other.
The use of digital certificates and passports are just two examples of many tools for validating legitimate users and avoiding consequences such as identity theft. What others exist?

Personally, I use PGP desktop these days for PGP. A screenshot is displayed below:
The version of PGP from MIT is horribly outdated. They state this themself:
On the other hand, the SRPMS (Source RPM - hence the source code) is available for RHEL 5.3 (which is what I use on my other laptops and prefer to Windows. This is a simple (relatively) build and compile process (assuming that you have GCC and the required libraries).

For the most part it is a simple process of:

./configure
make
make install

But again, this all depends on how your host is configured.

Now to authentication...

Exercise 16, Part 2

2009-07-21T09:49:00.002+10:00

Exercise 16, Part 2.

Visit 10 e-commerce websites. How many mention security on their home page? Is privacy mentioned? How many of them belong to the TRUSTe association?
Visit the Verisign web site - what solutions does it offer for e-commerce?
Visit your e-mail or WWW browser provider site and search for security. What technologies does your particular product support?
Visit the TRUSTe web site. Describe what services and solutions are offered.
Get the latest PGP software from http://web.mit.edu/network/pgp.html; install it on two machines and encrypt a message on one machine and decrypt it on the other.
The use of digital certificates and passports are just two examples of many tools for validating legitimate users and avoiding consequences such as identity theft. What others exist?

All of the sites I go to for ecommerce purposes (even the slightly dodgy ones) mention security. This does not mean they are secure, only that they espouse that they are.

Verisign is a all in one shop these days. They sell software, services and solutions. Ecommerce in a box. Some of these solutions and services are excellent (e.g. iDefense's offerings) other wane.

TRUSTe offers fluff plus. They talk about all they offer, but at the end of the day it does little if anything (I am biased).

As for the offerings of the web browser I am using (Mozilla in this case with No-Script enabled) click here for a separate post.

And Part 3...

Security Concerns and International Business

2009-07-21T09:40:00.003+10:00

Global e-commerce presents challenges exempt from domestic e-commerce. What security concerns add to the complexity of international e-business?

The primary concern is the disparity of international law. International provisions (such as EU Privacy regulations or SOX in the US) can impact orgainsations with no physical presece in a country that they trade with.

On top of this, IP (Intellectual Property) theft is made far simpler using the Internet. Many existing types of crimes can be replicated and transacted with the aid of an online environment. Further, novel new crimes designed to exploit the features and advantages of the Internet and other digital networks have emerged and are likely to continue to emerge in the future. Some examples of criminal activities that have benefited from the advances in digital technology include:

Computer break-ins (or Trespass) including the unauthorized admission to the whole or any element of a computer system without the right to do so;
Illegal interception without authority, created using technical methods of the non-public communication of computer data to, from or within a computer system;
Interference with or the damaging, deletion, deterioration, alteration or suppression of computer data without authorization;
Interfering with a system or the serious obstruction (without authority) of the execution of a computer system through the input, transmission, damage, deletion, deterioration, alteration or suppression of any and all electronically maintained data;
Possession of obscenity/prohibited pornography (e.g., child pornography and bestiality);
Industrial espionage;
Harassment;
Electronic Fraud (including email);
Web page defacements (cyber vandalism);
Theft of commercial documents.

While none of these crimes is wholly new, the ease in which they may be committed and the difficulty in capturing the offender has added a new dimension to crime. For instance, it is unlikely that law enforcement officials will be able to take action against many cyber-criminals unless the majority of countries first enact laws that criminalize the behavior of the offenders.
Some of the primary issues that face law enforcement in cybercrime cases include:

Increased Investigative Costs due to the need for high priced specialists;
The difficulties of conducting “Real Time” Investigations;
The ease of Anonymity on the Internet;
Difficulties with Jurisdictional issues;
The rate at which Technology is evolving; and
The Irrelevance of geographic distance.

Exercise 20: Modeling with UML

2009-07-20T23:16:00.002+10:00

Use Case, Class, Sequence, Collaboration, State chart, Activity, Component and Deployment diagrams are used in UML.

Describe each of the eight (8) main diagrams used in UML...

In my case I have completed subjects such as CSU's "ITC442 Object Modelling (8)" where the goal of the class was to:

The basic concepts of Object Orientation Systems are Class, Object and Message and these are studied extensively via diagrams, exercises and case studies. Object Diagrams are used to provide an introduction to the more abstract ideas of Classes and Inheritance. To study how an Object Orientation System works, messages are added to an Object Diagram to represent the processing of an example transaction. The main goal of the subject is to produce Class Diagrams that would be adequate to form the basis for further work in a computer project.

In this class a little while back I spent a fair amount of time on UML as this was the primary focus of the course. So rather than rehashing this or to even to make a summary of the many good resources on this topic (such as [1]), I have decided to go into how I do this process.

I Have first of all included a post on UML here.

I create UML and psudeo-code models when I reverse malware packers with work. To do this, I work in the opposite direction as to what many people would be used to. As such, I thought that I would approach this topic and exercise a little differently. I am going to go into running one of the tools I use, Altova UModel. This product is useful in creating models (and hence UML based case diagrams) from source code (and it is also useful when reversing binaries, but I am not covering that here in detail).
The tools can take source fragments and the parts you have reversed (using other tools - such as Rec Studio or even step by step with IDA Pro and OllyDebug) and create working models. There is a good deal of manual intervention, so you can not state that reversing is at all automated, but the UML tools do help.

The issue is that they do not do all languages. Altova does Java, VB, C#, but you also need other tools for C, C++ etc.

In addition, Mapforce (see the image below) can take UML basec case diagrams and create code (for those who do not want to do this themself) based on existing templates.
In my case, UML is a useful means of keeping track of what the program (in my case Malware or Packer) does when you are reversing it.

References:

[1] Bell, D. (2004) "UML's Sequence Diagram" IBM Software Group Technical Library. http://www.ibm.com/developerworks/rational/library/3101.html

Exercise 19: TP monitors and transaction protocols

2009-07-20T23:15:00.003+10:00

Exercise 19 has a few questions that I will answer from prior papers and books with a little tweaking. A little self plagiarism, but if you want to see the entire passages - buy my book :).

See and follow the link for a description in my own words of the ACID properties of a transaction.
Describe a TP monitor environment. How can a TP monitor stop an operating system being overwhelmed?
What is difference in load balancing with traditional and transactional MOM, RPC and conversations?
How can TP Monitors save money?
Why is a two-phase commit protocol better than a one-phase atomic commit protocol?
What is CICS by IBM?

Firstly, see the inline links for the other answers. Next and last, CICS by IBM is a:

"CICS (Customer Information Control System) is a family of application servers and connectors that provides industrial-strength, online transaction management and connectivity for mission-critical applications."

More simply put, it is a mainframe transaction processing solution that primarily runs on the IBM z/OS and z/VSE systems.

TP solutions are based on an architecture that is derived and consists of a TP-Monitor environment that is used to partition the workflow engine into a number of separate resource managers (RMs). Each of these is separately managed using a TP-Monitor environment. Together, the RMs together create a workflow engine that can be used to serve user requests.

By dividing the processing into discrete instances and running these on separate systems, individual hosts and servers are less likely to suffer from performance issues. This can save an organisation money in the long term. Better managed system provide the organisation with a means of controlling their systems. This allows better planning and lower downtime. These operational savings offset the increased hardware costs.

Exercise 18: Threading demonstration in Python

2009-07-20T23:13:00.001+10:00

Here we investigate a simple demonstration of the threading module in Python that uses both a lock and semaphore to control concurrency is by Ted Herman at the University of Iowa.

This exercise was looking at a python demo script. My preferred platform is RHEL and many security tools are python based, so this was not anything particularly new...

Exercise 17

2009-07-20T23:11:00.004+10:00

Here we add definitions for eight terms and concepts used in threaded programming:
1. Thread Synchronisation
2. Locks
3. Deadlock
4. Semaphores
5. Mutex (mutual exclusion)
6. Thread
7. Event
8. Waitable timer.

Follow the links.

Malware Analysis

2009-07-20T21:58:00.003+10:00

In order to really get to the roots of malware (this is Trojans, virii, etc) we need to perform in-depth analysis of malware samples, including capabilities, structure, and map the relationships to other samples and use these to train our inputs. This has been the focus of much of my research.

To define the issue, “Malware” varietals include virus, worm, trojan, and backdoor programs.

The way I have been researching this is in the creation of perceptrons that utilize a combination of static and dynamic analysis techniques. Such a process is defined in the figure below:
Here, we go through the following stages:

Pre-Processing, this is the process used to process the raw packet data, memory segments etc. This process can be defined using existing technologies (e.g. Snort for packet capture, BPFs etc.),
Feature Construction. In this phase we construct the model statistics. This is we take the basic features of the data and extract values that could be of use,
Apply Algorithims. Here we use rule learnign techniques to find the malware.

This is an iterative process. This involves both behavioural and static analysis of code with the primary constraint being time. Users want to access their systems quickly. If the process is too slow, accuracy will not matter as the user will find a way to bypass it and subvert the protection.

References:
Wenke Lee, Sal Stolfo, & Kui Mok., (1999)
“A Data Mining Framework for Building Intrusion Detection Models”,
Proceedings of the 1999 IEEE Symposium on Security and Privacy, Oakland, CA, May 1999

Exercise 15: Part 7 - 10

2009-07-20T21:40:00.003+10:00

Download a virus checker and read the documentation.

I am contracted to a large anti-malware firm to write unpacker code. I will skip this one.

As for the operation, the AV engine unpacks the malware and checks it against a signature base. This is then checked against set heuristic behaviors.

I will not go into detail in this post, but will add a separate post on this topic.

Exercise 15: Part 5

2009-07-20T21:36:00.001+10:00

Find out if your university or workplace has a backup policy in place. Is it followed and enforced?

Yes, we have both a GFS (grandfather father son) and an online mirror.

This is as we value information.

Exercise 15: Part 4

2009-07-20T21:31:00.002+10:00

Does the company you work for (or the school you attend) utilise a proxy server for Internet access? Is the proxy server intended to keep hackers out of the network, or control employees’ access to the Internet?

Both of my companies (Information Defense and Integyrs) use a Proxy server. We use SQUID. We have moved from ISA from Microsoft. These are for Web traffic access. They are not used for may of the otherInternet protocols (although some Proxies are available for this reason).

These are not for "keeping out hackers" etc. They are a monitoring and measurement device with an added feature of some cached information.

We have a dual Cisco/IPTables Firewall system with inline devices and a combination of OSSIM and SNORT IDS with some IPS functionality. If it makes the Proxy, it has done all it is going to do.

Exercise 15: Part 3

2009-07-20T21:28:00.001+10:00

Accessing a firewall vendor site, find out what solutions are offered?

In order to define what we can get from a firewall, we first we need to define a firewall. A firewall is an application, device, system, or a group of systems that controls the flow of traffic between two networks based on a set of rules, protects systems from external (internet) as well as internal threats, separates a sensitive are of a private network from less sensitive areas, encrypts internal and external networks that transmit sensitive data, or hides internal network addresses from external networks (Network Address Translation). A firewall picks up where the border router leaves off and makes a much more thorough pass at filtering traffic. It comes in different types, including static packet filters (ex. Nortel Accelllar router), statefull firewalls (e.g. Cisco PIX), and proxy firewalls (e.g. Secure Computing Sidewinder).

Similar to routers, a firewall uses different filtering technologies or methods to ensure security. These methods include packet filtering, statefull inspection, proxy or application gateway, and deep packet inspection. A firewall can use just one of these methods, or it can combine different methods to produce the most appropriate and robust configuration.

I have not added quotes, as this is taken from a few of my own published papers and books.

Exercise 15: Part 2

2009-07-20T21:20:00.002+10:00

What makes a firewall a good security investment? Accessing the Internet, find two or three firewall vendors. Do they provide hardware, software or both?

For vendors, there are today many. Firewalls are a mature market. Some of these include:

FWSM,
ipfilter,
ipfw,
iptables,
PF,
Cisco PIX (now really ASA, but I am showing my age);

I helped write a Checkpoint book, so I will plug them here as well.

Many vendors (e.g. Cisco) focus on Hardware, but others (including GPL'd software such as IPTables) are solely software based.

Simple put, a good firewall is one that gives a positive return on investment. This is measured by testing.

This comes from the firewalls ability to maintain and monitor security with a low level of maintance. This is all about the configuration of the firewall platform itself. All firewalls have an operating system (OS). Do not be fooled by vendor assertions that they have an appliance. Firewalls and routers are all software driven; all they do is make it more difficult to see the code. Next it is important to ensure good practice occurs in respect of system administration (user management, patch updates, change control and configuration backups). If the firewall is not patched it will eventually be compromised. Just as it is a security device does not make it automatically secure. Finally, it is necessary to validate that the firewall rulebase matches the organizational policy.

Testing the firewall should be coordinated with testing the other components of the organization’s defense-in-depth methodology. The organization should not only rely on a single line of defense otherwise this is a red flag that needs to be raised. Firewalls are not the panacea for all security ills. They slow attackers and log.

The overall result of the testing or audit of the firewall would be the identification of any security vulnerabilities as well as an assessment of whether the firewall is fulfilling its function in relation to the security policy of the company. An assessment should be made on whether the set up, configuration and operation of the firewall is secured sufficiently to protect the information or services that it is intended to guard, considering the risks that were identified and the likelihood of occurrence.

Center for Internet Security also provides benchmarks for several specific brands of firewalls devices. The benchmarks (available at http://www.cisecurity.org) greatly aid in developing an audit program for firewalls. These are the source of our checklist frameworks.

Exercise 15: Part 1

2009-07-20T21:10:00.002+10:00

Can a simple firewall be designed from standard computer equipment? What hardware components would you need for a proxy server?

The answer to this is and for the last 20 years has been of course. It is also how long is a piece of string for it all comes to what you need it for, this is correct and adequate provisioning.

There are many Proxies from the venerable Netscape of the past to some of the more common modern proxies -

Microsoft Proxy (now a part of the Forefront Suit of Products)
Squid (Linux to begin, but now on Windows as well).
Tiny Proxy

In fact, even Apache can run as a Proxy. Linux comes with support for IPTables, a simple stateful firewall.

Since the fundamental purpose of the firewall is to manage the flow of information between two networks, it by nature must provide such function through a detailed configurationthat filters or allows traffic to pass through as consistent with the security policy. In developing a firewall, some of the critical things to look at are as to whether:

The access rules (authentication, authorization and accounting) for the firewall is in line with the security policy and best practices
access to the firewall system for management and maintenance is provided using an encrypted channel
physical access to the device is restricted
the firewall is configured to hide internal restricted DNS information from external networks
external firewall restricts incoming SNMP queries
the firewall is configured as “fail closed”
the firewall hides the internal information from external sources
the firewall is configured to “deny all services, unless explicitly allowed”
all security–related patches are applied to the firewall system
configuration settings are properly backed up and accessible to authorized personnel only

Exercise 15: Review questions

2009-07-20T21:04:00.003+10:00

Today we have a number of smaller questions to answer. These are:

Can a simple firewall be designed from standard computer equipment? What hardware components would you need for a proxy server?
What makes a firewall a good security investment? Accessing the Internet, find two or three firewall vendors. Do they provide hardware, software or both?
Accessing a firewall vendor site, find out what solutions are offered:

http://www.checkpoint.com

http://www.microsoft.com/catalog/display.asp?subid=22&site=10538&x=44&y=21

Does the company you work for (or the school you attend) utilise a proxy server for Internet access? Is the proxy server intended to keep hackers out of the network, or control employees’ access to the Internet?
Find out if your university or workplace has a backup policy in place. Is it followed and enforced?
Most of the antivirus software perform an active scanning of the user activity on the Internet, detecting downloads and attachments in e-mails. Hackers have readily available resources to create new viruses. How easy is it to find a virus writing kit? Search the Internet and find such a tool. For example, see what you can find at http://vx.netlux.org/dat/vct.shtml.
Download a virus checker and read the documentation.
How does it operate?
What is the process of updating the virus signature file?
How does the publisher charge for the product/service?

Click the question itself to go to the answers.

For question 6 and other parts of this question, I shall write a separate post. The answer is that it is simple to find malware creation tools. The issue is that these are the easier ones to find as part signatures are still signatures. The issue is when a completely new code is published.

Unexpected User Input

2009-07-20T20:56:00.002+10:00

No matter where a page or who you believe your clients to be, you cannot trust any information that is sent to website. All import to any application or database must always be sanitized. It may be possible to limit the size of restrictions on import rules, but they need to exist.

Additionally, as we have noted above is essential to send any information that you require to remain private using POST and not GET methods. Were special characters are sent to the Web server they should be stripped or escaped and if necessary mapped to an alternative but safe character.

Next, users do not need detailed output and diagnostics. This may be necessary for development teams but it is not advisable to send this information to the client. If an unhandled error occurs send a generic error page. Customized pages allow an attacker to find out what exactly went wrong and to tailor their attacks.

If you are sending sensitive information always use encryption. This includes both information from the user and returns from the server. Additionally, although these methods may be bypassed by an attacker, some anti-caching techniques should be included where possible.

Input validation
Good practice within put validation requires that you trust nothing from the client. Most security people understand this aspect but what they also forget is that you should also trust nothing from your own database. The Web application should only trust that information which is explicitly set and loaded into the application itself.

Sanitization
Anything at all that is received through a Web front end or application should be validated. Any event that there are any potentially unsafe characters, regular expressions and filters should be used to sanitize this information. This requires either deleting the offending characters or re-mapping them to an alternative but safe alternative.

Error checking
When designing a site in need to contend with errors from the website, database, application and the network itself. It is generally easy to predict certain errors; the problem is that many errors will be unexpected. Error checking needs to handle all possible conditions and if an uncalled for event occurs the error needs to be sent in explicit detail to the site administrators or returning a generic error message to the user who experienced the error.

Web Browser Security
There is little that most sites can do to secure the web browsers used by their clients. In the event that a Web application is running internally (or even to protect your own users browsing experience), the following site presents an introductory methodology that will aid in protecting many of the attacks against your systems. When talking about internal Web browsing is the responsibility of the organization to ensure the security of their clients systems. The methodology used by Cert in the following site goes long way to achieving this goal.

See:

http://www.cert.org/tech_tips/securing_browser

REMEMBER
Your clients can chnage ANY value they recieve or send. Never trust a web broswer. There are MANY ways to interect (e.g. a Proxy) and modify any value that comes from a client.

Session tracking and management

2009-07-20T20:53:00.001+10:00

As I have noted in other posts, GET is problematic when it comes to managing ongoing sessions.

Apart from the character length limitations there are security concerns. Using PUT is not without its own issues. Entity authentication is generally a problem and this is reflected within the OWASP Top 10 (the 2007 top 10 list rated this as number seven). Are the only way to solve this issue is to ensure secure communication and credential storage and creation.

This begins with the generation of strong session IDs are based on random values. Incrementing values in a sequential manner is a sure way to allow attackers to determine your security model. OWASP (http://www.owasp.org/index.php/Top_10_2007-A7) contains numerous considerations that should be addressed when considering the manning session management.

Session Tokens
A number of important issues concerning session management are listed in the section below.

Cryptographic Algorithms for Session Tokens
Always create session tokens that are user unique, non-predictable, and resistant to reverse engineering. Utilise a trusted source of randomness when creating the token (a pseudo-random number generator is recommended). Session tokens need to be associated with a particular HTTP client instance to prevent hijacking and replay attacks.

Appropriate Key Space
The token's key space should be suitably sufficient to prevent brute force attacks.

Session Time-out
Session tokens that are not set to expire at the HTTP server can permit an attacker guess or brute force a valid authenticated session token indefinitely. In the event that a user's cookie is intercepted or brute-forced, an attacker could use static-session tokens to access to that user's accounts. Session tokens can also be captured by logging and proxy cache servers. Any which could be compromised leading to a compromise of many accounts.

Regeneration of Session Tokens
It is recommended that the HTTP server automatically expire and regenerate tokens. If set up frequently enough this will reduce the time window in which an attacker can instigate a replay exploit if they capture a legitimate token. This will aid in preventing social hijacking and brute force attacks against an active session.

Session Forging/Brute-Forcing Detection and/or Lockout
A session token brute-force attack can allow an attacker to attempt sending many thousand possible session tokens that embedded in a legitimate URL or cookie. Intrusion-detection systems frequently do not alert on this type of attack and it is commonly overlooked during penetration tests. The answer to this is a requirement to lock out accounts. Web developers should implement honey cookies (these are "booby trapped" session tokens that a desire not to be assigned to a user but will send out alerts if they used).

Session Re-Authentication
Considerate re-authenticate users for critical actions like as money transfers or purchases involving considerable sums of money. This would require the user to re-authenticate or be reissued another session token directly preceding taking a material action.

Session Token Transmission
In the event that a session token is captured using a network sniffer, a web application account is then extremely vulnerable to a replay or hijacking attack. Always use web encryption technologies (such as Secure Sockets Layer (SSLv2/v3) and Transport Layer Security (TLS v1) protocols) to protect the state mechanism token in transit.

Session Tokens on Logout
Many users will access the site from an insecure computer (such as an Internet Kiosk). For this reason it is a good idea to have the application overwrite and destroy session cookies either when the user logs out of the application or after a timeout period.

Page Tokens
Page specific tokens or "nonces" should be used in combination with session specific tokens. This helps provide an additional measure of protection for client requests. Page tokens can be used to reduce the impact of MITM (monkey/man in the middle attacks). Page tokens can be stored in cookies or query strings. It is essential that an effective means of randomized in the values used on a page token is employed. Page tokens also make brute force session attacks more difficult.

Exercise 16

2009-07-02T20:42:00.003+10:00

Answering exercise 16 we have the following:

Visit an e-commerce website and survey the mode of payment allowed. Would you trust the site with your business?
Global e-commerce presents challenges exempt from domestic e-commerce. What security concerns add to the complexity of international e-business?
What measures should e-commerce provide to create trust among their potential customers? What measures can be verified by the customer?
Visit 10 e-commerce websites. How many mention security on their home page? Is privacy mentioned? How many of them belong to the TRUSTe association?
Visit the Verisign web site - what solutions does it offer for e-commerce?
Visit your e-mail or WWW browser provider site and search for security. What technologies does your particular product support?
Visit the TRUSTe web site. Describe what services and solutions are offered.
Get the latest PGP software from http://web.mit.edu/network/pgp.html; install it on two machines and encrypt a message on one machine and decrypt it on the other.
The use of digital certificates and passports are just two examples of many tools for validating legitimate users and avoiding consequences such as identity theft. What others exist?

To answer these, I started by going to EBay. I registered an account, supplying a pile of useless information and going through "captures" that can be easily deciphered by computers (usually quicker than by humans these days). It took around 4 tries to have eBay happy with the information, including having to make some up in fields that would otherwise not apply to me.

I also had to run a number of scripts. I was not happy with this. I run no-script in Mozilla. Then to the email, a simple clear text means of intercepting user accounts. Too bad so many firms do this REALLY poor method of validation for e-commerce.

Next it is the wait for the code. A long convoluted process that does nothing but fool people into a false belief that this has done ANYTHING to help secure your site.

So far, I have logged into eBay and I am still in an SSL session. but wait, I click to search or look at an item and I drop to plain HTTP. This is I lose the encryption. My session data is still being sent, so not I can have a session hijack take over my information. My bidding it all clear text. This means it can be intercepted.

So, would I use eBay, No! Nor do I. There are MANY far better and more secure auction sites that this popular but low in the heap provider.

Trust really needs to be earnt. Many sites (even high end well known ones such as eBay) pray on the ignorance of their users. Rather than creating the perception of security, what an organisation should do is to create a safe site.

The purpose of information security is to preserve:

Confidentiality - Data is only accessed by those with the right to view the data.
Integrity - Data can be relied upon to be accurate and processed correctly.
Availability -Data can be accessed when needed.

Consequently, the securing of information and thus the role of the Security professional requires the following tasks to be completed in a competent manner:

The definition and maintenance of security policies/strategies.
Implementing and ensuring compliance to Policies and Procedures within the organization:

a. The IT security organization needs a clear statement of mission and strategy. Definition of security roles & processes.

b. Users, administrators and managers should have clearly defined roles/responsibilities and aware of them.

c. User / support staff may require training to be able to assume the responsibilities assigned to them.

Effective use of mechanisms and controls to enforce security
Well defined Technical Guidelines and controls for the systems used within the organization
Assurance (audits and regular risk assessments).

IT security is not about making a perfect system, it is about making a system that is resilient and that can survive the rigors it is exposed to. Compliance comes down to due diligence. If you can show that your system is resilient to attack and that it has a baseline of acceptable controls, you will be compliant with nearly any standard or regulation. This will help build trust. The idea is not just to meet the minimum, but to actually show due care.

On to part 2...

Exercise 14

2009-07-02T20:41:00.004+10:00

1. You can learn more about Cookies at: http://home.netscape.com/newsref/std/cookie_spec.html

Even better, I have my own primer on cookies (and a few other posts) at
http://gse-compliance.blogspot.com/2007/12/cookie-primer.html

2. You can learn more about electronic payment systems with Reading 9 on Electronic Reserve on the CSU Library Service Web site at: http://www.csu.edu.au/division/library/eservices/ereserve.htm

The article is by Schneider, P & Perry, JT 2001, ‘Electronic payment systems’, Chapter 7, in Electronic Commerce, Course Technology, Boston.

We have moved a long way from this model...

Exercise 15

2009-07-02T20:41:00.002+10:00

1. Can a simple firewall be designed from standard computer equipment? What hardware components would you need for a proxy server?

2. What makes a firewall a good security investment? Accessing the Internet, find two or three firewall vendors. Do they provide hardware, software or both?

3. Accessing a firewall vendor site, find out what solutions are offered:
http://www.checkpoint.com
http://www.microsoft.com/catalog/display.asp?subid=22&site=10538&x=44&y=21

4. Does the company you work for (or the school you attend) utilise a proxy server for Internet access? Is the proxy server intended to keep hackers out of the network, or control employees’ access to the Internet?

5. Find out if your university or workplace has a backup policy in place. Is it followed and enforced?

Exercise 13

2009-07-02T20:39:00.002+10:00

1. List and describe your experiences with a secure Web site. Begin by defining some security issues in Reading 8. Some examples may be:

University enrollment;
online banking, auctions, real estate;
booking a cheap air ticket or concert ticket;
shopping online for a book, software or a CD.

2. What is SET and how does it compare to SSL as a platform for secure electronic transaction? Is SET in common use?

First, see exercise 12. SET is long dead and all that remains is for it to be buried.

As for my experiences with "secure" web sites, the first issue is the terminology itself. SSL is not secure, it is an encrypted tunnel, one which can be intercepted (as we rarely use client side certificates). Strangely enough, the ASIC (Australian Securities and Investment Corp) has one of the better secure sites. The signing site for company directors is fairly well designed. Something unusual for a .gov organisation.

This site uses a client certificate for authentication. This allows the user to lodge company statements minus the MiTM attack problem. There is an issue of protecting the certificate that is not addressed and which is left to the client, but nothing is perfect.

The links below address a few issues with secure site design:

What is really essential to remember is that it matters what the client can see and alter. A web browser is NOT secure and either a client or an attacker can intercept and change values going to and from a web session.

Exercise 12: Designing for a secure framework

2009-07-02T20:33:00.004+10:00

Find out about SET and the use of RSA 128-bit encryption for e-commerce.
Design a Web application form for a new credit card.
What can you find out about network and host-based intrusion detection systems?

I have to state that SET is the walking dead. It was raised by Visa, Mastercard and a few other card makers in the late 1990's and died a slow painful death. It is the walking dead as they keep trying to revive it.

Just like the walking dread from one of Lovecraft's novels, SET is a walking corpse. Setco has gone a long time ago and there remains little but the smell.

Point 2 (and this is the second time I am writing this as Windows has again blue-screened due to issues with drivers in a typical Microsoft poorly conceived and executed manner), was conducted in exercise 9 fro accepting cards. This was far from ideal and secure and did not meet many of the requirements of a PCI based site, but it did take cards. The process to actually apply is far more complex (and to some extent more complex than it needs to be). Having just completed a process that required a 1 minute phone call and a single fax'd signiture, I was suppriused at the OnLine alternative that I co9uld have used (over around 30 minutes). The online process needs to be as simple as the paper one.

Part 3 is where I have really spent my time and effort. I have been working on databases for both host and network based Intrusion Detection systems and in fact an integrated solution.

This process has involved the basic Snort MySQL dataset which I have been including IPTables logs and system logs into. I manage the database using XCase. This is an tool that allows me to manage the ERD (or entity relationship diagram) and hence the associations between the tables.
As you can see from the daigram above, there are a large number of tables that I am in te process of linking. These match attack signitures by default. The idea is to use simple neural and learning networks to match associations and improve the results - hence minimising false negatives and wasted time on false positives.

For the most part, network based IDS (see www.snort.org) or host based IDS (see Tripwire) are based on signiture matching. This is slowly changing.

Assessment Item 1

2009-06-21T15:00:00.001+10:00

For this subject we have an ongoing assessment schedule. This can made for some hectic periods (with work and all) but it does mean that you have to focus on the topic over the entire period and not simply cram.

The first assessment item for this subject is the combination of the Exercises for the initial six (6) topics, the first four (4) Ruby language Workshops and an elevator pitch. These items are linked below.

Exercises 1-11 (of 26)

Ruby on Rails Workshops 1-8

Elevator pitch No. 1

Elevator pitch 1 on progress report for assessment item 1

For the next week I am not likely to post anything at all.

Ruby on Rails - Workshop 4 (Solution - Part 2)

2009-06-21T14:54:00.009+10:00

Solution...
Here is how to do this workshop. The results are posted in the sections below.

3. Compare the Ruby and Python versions of the dog years calculator:
#!/usr/bin/ruby
# The Dog year calculator program called dogyears.rb

def dogyears
# get the original age
puts “Enter your age (in human years): "
age = gets # gets is a method for input from keyboard
puts # is a method or operator for screen output

#do some range checking, then print result
if age <> 110
puts "Frankly, I don't believe you."
else
puts "That's", age*7, "in dog years."
end
dogyears

Python

#!/usr/bin/python
# The Dog year calculator program called dogyears.py

def dogyears():
# get the original age
age = input("Enter your age (in human years): ")
print # print a blank line

# do some range checking, then print result
if age <> 110:
print "Frankly, I don't believe you."
else:
print "That's", age*7, "in dog years."

### pause for Return key (so window doesn't disappear)
raw_input('press Return>')

def main():
dogyears()
main()

The most obvious difference is the interpreter used. Ruby and Python have their own interpreters and of course point to these respectively.
We have the following differences:

Ruby	Python
Keyboard gets ______________ if-else : is optional ______________ keyword elsif ______________ Main Not necessary ______________ Output puts ______________	Input input() ______________ if else : is required ______________ keyword elif ______________ function Is required ______________ Output print() ______________

There are more differences (e.g. Ruby calls "Def dogyears" and Python requires "Def dogyears()" with the brackets following the function call).

This will do for now though.

Click to return to the main page of this Workshop.

Ruby on Rails - Workshop 3 (Solution) - Part 2

2009-06-21T12:57:00.005+10:00

This is Part 2 of the Workshop 3 solution. Click here to see part 1 of the solution.

My preference is to use "DbVisualiser" (below) or XCase - this is instead of the MySQL tools (listed in part 1 of the workshop). I use MS SQL and Oracle databases more then MySQL, so I want a tool that connects to many types of database server. These tools also have a reverse engineering feature to map existing databases and create schema diagrams.
Step 1
First, let's create the OTBS database using DbVisualiser (we have already logged into the database before using our new credentials).
Then we add the table and column.
Which gives us a simple database.
Now we have a database that we can link into our web application. In reality and in a real situation, we should really look at the table a little more. I have guessed (poorly) the column types and lengths. These contraints and the data fields should be configured better than this.

SQL Database attacks (e.g. SQL injection) are one of the most common attacks today. Ruby does not protect you from this. This is the job of the coder to make secure code. We really should be starting to think of this in advance and teach this.

Step 2
Lets go back to our Passenger directory. Using the data from the workshop, we first use the following command:

ruby script/generate scaffold Passenger name:string job_id:integer contact_number:string suburb_origin:string street:string street_number:string building:string suburb_destination:string passenger_number:string taxi_type:string date:string time_required:string

As can also be seen in the command line image below:
This creates our scaffold in Ruby.

Alternatively, with the database defined (see step 1 above) we can simplly issue the command:

ruby script/generate model Passenger

This is not needed in our case and you should only do this in one way (not create and re-create). But I just wanted to demonstrate that there are multiple ways that you can create your model.

Now we have our tables and a small ruby web server. Next we start in integrate these and make something a little more useful.

At this point I have gone back and tidied the database a little. I have set constraints on a few fields.

taxi_type - set to only the following options (sedan, wagon, disabled, maxi)
passenger_number - set to allow values 1 to 4

This is Part 2 of the Workshop 3 solution.
Click here to see part 1 of the solution.

Ruby on Rails - Workshop 4 (Solution)

2009-06-21T11:25:00.002+10:00

Solution...
Here is how to do this workshop. The results are posted in the sections below.

1. Create, test and debug a Ruby program called dognames.rb or catnames.rb to accept 3 names from the keyboard and to display each name on the screen in alphabetical order WITHOUT using a data structure such as a list.

We start with the code that requires input (the dogyears) and have a play:
Then we code up a small program to do the required steps (notice this is still not securely coded):
And running this we see:
Nothing really pretty.

Vince Cheng did this in another subject and has a different code sample that also works.

2. Write a Ruby program called fizzbuzz.rb that prints the numbers from 1 to 100. But for multiples of three print "Fizz" instead of the number and for the multiples of five print "Buzz". For numbers which are multiples of both three and five print "FizzBuzz".

Being that we have a working copy of this, I just played with the existing code.
Which, as you can see works correctly:
So on to the last part of Workshop 4...

This workshop is continued on another page. Click here to see part 3 of the Workshop 4 challenge question.

Click to return to the main page of this Workshop.

Ruby on Rails - Workshop 3 (Solution)

2009-06-21T11:24:00.002+10:00

Solution...
Here is how to do this workshop. The results are posted in the sections below.

Step 1 Before we even start doing anything else we need to stop and take action on a step NOT included in the workshop.
This is security!

Let us do this the easy way. Start by clicking on the Instant Rails management app.

From here, lets use the "PhPMyAdmin" configuration tool. Click this and the local web configuration page will open.
Notice that the server is running as user "root" with no password - BAD RUBY!!! This joke that you are safe as the MySQL is locked to localhost is a joke that should be squashed. The Instant Rails developers should be ashamed of themselves for this criminal lack of wisdom.

Just stating that it is linked only to localhost shows a true lack of any security awareness.
So, click on the "privilages" tab and lets fix this horiffic flaw from those uncaring Rails developers who are seeking to destroy your site security.

Here we need to add other users and give them a password. I will not go into details here, but do NOT leave this without a password!

Select root (I will not go into setting up many users, but you should do this) and click the "Edit Privilages" link (as is circled in red below).

This will take you to the use privileges page. Scroll down to the change password field.This field is set by default to "No Password" - Bad Instant Rails. Add a password and click "Go".
You should see the confirmation message stating the that password has been successfully updated.
Step 2
Now you can log into the MySQL server. Open the MySQL Query browser.

Note that you will need to add both the username and a password (this is the one we configured in step 1).
Clicking "ok" will log us into the MySQL Query browser.

We will continue this in another post.

Click here to continue to Part 2 of Workshop 3.

Ruby on Rails - Workshop 2

2009-06-21T10:01:00.003+10:00

Solution...
Here is how to do this workshop. The results are posted in the sections below.

Step 1
First, we need to install and configure Rails. We installed rails in workshop number 1. In this workshop we get it running. So let us start by starting Rails and having ti run.This starts our Apache server and MySQL (lite) server running on the local host. All of the configuration steps have been completed in workshop 1, so this is just a restart of the application.

Step 2
Next we go to the Rails directory with our command line.
We run "use_ruby.cmd" from the command line and set the path and other environment variables.

Step 3
From this command line (with the environment set as in step 2) we can create our project. I have selected the directory name "rails_projects" (not a great deal of creativity here).

Next, we let rails create the project environment. To do this we run the command:
"rails Passenger" from the projects directory.
Rails automatically creates the required directory structure and populates the basic strcutres used. We can see this for Windows Explorer (below).
This is still rather empty, but we have not actually done anything other than raise the project framework as yet.

Step 4
Next we change directory to our project folder that was created in the step above.
Running the command, "ruby script/server Webrick" starts the server and project running and waiting for requests. This is set to serve requests on TCP 3000.

Step 5
Now, when we connect to the local ruby web server on TCP port 3000 we get the unpopulated webpage.
This of course has little content (other than the default ruby on rails framework). We can test that it is working. Clicking on the link "About your application's environment" is a simple ruby script we can use to test the server.
We can see from the above screenshot that we have received and processes a request.
And as such, the information table listing our envirioment is displayed.

In Workshop number 3 we shall start to populate the database.

Workshop 2 - Question on Convention

2009-06-20T13:07:00.002+10:00

What is meant by “convention over configuration” and how does it reduce coding?

Convention over configuration (CoC) is a "Design a framework so that it enforces standard naming conventions for mapping classes to resources or events. A programmer only needs to write the mapping configurations when the naming convention fails".

Through a simplification of the objects and designations used when coding, the developer can spend more time coding and less in trying to remember what something was named.

BuildingWebApp, Why You Should Learn Ruby on Rails, What is Ruby on Rails, viewed by 17 June 2009 http://www.buildingwebapps.com/podcasts/6514-why-you-should-learn-ruby-on/transcript

Workshop 1 - Challenge 4

2009-06-20T12:06:00.012+10:00

What is meant by “convention over configuration” in regards to the use of Rails in Web application development?

Convention over configuration (or coding by convention) is the design paradigm used by Ruby in order to decrease the volume of decisions that developers are required to make. This is aimed at creating a simplified coding platform. This can result in a loss of flexibility.

[1]Curt Hibbs (2005) "What Is Ruby on Rails" (10/13/2005) http://www.onlamp.com/pub/a/onlamp/2005/10/13/what_is_rails.html, O'Reilly, OnLamp.

Workshop 1 - Challenge 6

2009-06-20T12:06:00.010+10:00

Describe the steps involved with the MVC design approach.

The steps involved with the MVC design approach can be represented by the MVC communication cycle. In general, this cycle begins with an input from the user. This can be seen in the representation of this process sourced from "Moock" below:

We can also see "the relationships between the model, view, and controller layers of an MVC application" in the depiction from Sun MicroSystems in "Designing Enterprise Applications
with the J2EE Platform".

Figure 11.1 The Model-View-Controller Architecture

As such, the process is as follows.

First, the view receives user input and passes it to the controller,
Next, the controller receives user input from the view,
The controller then modifies the model in response to user's input,
The model will then change when it receives an update from the controller,
The model subsequently sends a notification of the change to the view,
Whereas the view will proceed to update the user interface

There are also cases where the controller can update the view directly.

References:
[1] Glenn E. Krasner & Stephen T. Pope (1988) "A Description of the Model-View-Controller User Interface Paradigm in the Smalltalk-80 System" ParcPlace Systems, Inc. 1550 Plymouth Street Mountain View, CA 94043 glenn@ParcPlace.com
[2] Ralph E. Johnson. (1987) "Model/View/Controller" Department of C.S., U. of Illinois, Urbana-Champaign, November, 1987
[3] Steve Burbeck, (1987) "Applications Programming in Smalltalk-80(TM): How to use Model-View-Controller (MVC)" Online at http://st-www.cs.uiuc.edu/users/smarch/st-docs/mvc.html, viewed 20th Jun 2009.
[4] Trygve M. H. Reenskaug (1978) "MVC" http://heim.ifi.uio.no/~trygver/themes/mvc/mvc-index.html, viewed 19th June 2009

Workshop 1 - Challenge 5

2009-06-20T12:06:00.005+10:00

When did Model-View-Controller begin and where is it used?

The MVC paradigm owes its origins to Smalltalk. As was noted by Krasner & Pope (1988):

Model-View-Controller (MVC) programming is the application of this three-way factoring, whereby objects of different classes take over the operations related to the application domain (the model), the display of the application's state (the view), and the user interaction with the model and the view (the controller). In earlier Smalltalk system user interfaces, the tools that were put into the interface tended to consist of arrangements of four basic viewing idioms: paragraphs of text, lists of text (menus), choice "buttons," and graphical forms (bit- or pixel-maps). These tools also tended to use three basic user interaction paradigms: browsing, inspecting and editing.

The MVC method has grown since Smalltalk with large organizations (including Microsoft) promoting it. The MVC is an architectural paradigm within the field of software engineering that is designed to isolate business logic from those considerations related to the user interface.

The first concepts of the MVC framework developed in the early 1970's in a series of papers on modeling shipyard processes (1973). When the author of these papers (Reenskaug, 1978) moved to XEROX PARC in 1978, this concept was incorportated into the SmallTalk code model and was developed into the MVC framework we have today.

References:
[1] Glenn E. Krasner & Stephen T. Pope (1988) "A Description of the Model-View-Controller User Interface Paradigm in the Smalltalk-80 System" ParcPlace Systems, Inc. 1550 Plymouth Street Mountain View, CA 94043 glenn@ParcPlace.com
[2] Ralph E. Johnson. (1987) "Model/View/Controller" Department of C.S., U. of Illinois, Urbana-Champaign, November, 1987
[3] Steve Burbeck, (1987) "Applications Programming in Smalltalk-80(TM): How to use Model-View-Controller (MVC)" Online at http://st-www.cs.uiuc.edu/users/smarch/st-docs/mvc.html, viewed 20th Jun 2009.
[4] Trygve M. H. Reenskaug (1978) "MVC" http://heim.ifi.uio.no/~trygver/themes/mvc/mvc-index.html, viewed 19th June 2009

Workshop 1 - Challenge 3

2009-06-20T12:05:00.002+10:00

What is Rails and how does it work with Ruby?

Rails is a framework for the Ruby language. Rails simplifies working with Ruby (see the following link for some issues with doing it yourself).

Hibbs (2005) defines this suscintly in that "Ruby on Rails is a web application framework written in Ruby, a dynamically typed programming language similar to Python, Smalltalk, and Perl".

[1]Curt Hibbs (2005) "What Is Ruby on Rails" (10/13/2005) http://www.onlamp.com/pub/a/onlamp/2005/10/13/what_is_rails.html, O'Reilly, OnLamp.

Workshop 1 - Challenge 2

2009-06-20T12:04:00.002+10:00

Ruby is “an interpreted scripting language” for quick and easy object-oriented programming”. Find out about the Ruby language and discover what this means.

For the first part of the answer see this link.

The definition of "object-oriented programming" (OOP) is listed at Sun's Java Tutorial site. This site states that:

An object is a software bundle of related state and behavior. Software objects are often used to model the real-world objects that you find in everyday life. This lesson explains how state and behavior are represented within an object, introduces the concept of data encapsulation, and explains the benefits of designing your software in this manner.

As a result, OOP is a programming language model that has been developed using "objects" in place of "actions". This results in an OOP being based on a focus of "data" in place of "logic".

See http://searchsoa.techtarget.com/generic/0,295582,sid26_gci1148576,00.html?offer=briefcase

Ron Lynam

2009-06-19T08:34:00.002+10:00

Last night my grandfather died at the age of 93. This follows a long battle with Parkinson's disease.

He will be sorely missed by family and friends.

My grandfather was an electronics expert and engineer in the early days of this discipline. It was through him I gained a love and passion for computers and electronics. It was because of him that I first learnt to program. He taught me Assembly and C (which at the time I generally used to reverse engineer games such that my sisters could never win).

It was through him that I obtained my first Unix terminal account on a dial up modem (75/300 baud) in 1979. As a consequence, it is him I owe my current career to.

Pop, I will miss you.

Exercises down

2009-06-17T18:21:00.002+10:00

Well that is Exercises 1-11. I have the elevator pitch and exercises complete. Now to complete the workshops and load the recording of the pitch...

Now if I can just find a way to make a 40 hour day...

Exercise 11: Part 3

2009-06-17T17:39:00.001+10:00

What are DOMs and why were they developed?

Document Object Models (or DOMs) are a means of representing the concepts that are associated with and incorporated into the contents of that document.

W3C notes that a DOM is a "platform- and language-neutral interface that will allow programs and scripts to dynamically access and update the content, structure and style of documents. The document can be further processed and the results of that processing can be incorporated back into the presented page. This is an overview of DOM-related materials here at W3C and around the web."

The primary reason for the existence of DOMs is the creation of some level of standardization. It may seem great to have everyone doing their own thing, but think where that could lead us...

If each browser included its own new HTML tags (as occurred in the early days of the web), its own scripting language etc, we end up being left with the need to create separate sites for any and all browsers that may connect.

.

Exercise 11: Part 2

2009-06-17T17:36:00.002+10:00

XML schema is a forthcoming development of the technology. Visit the W3C website and search for information on schema. What are the benefits of adopting a schema standardized for a business sector?

Ogbuji (2005) notes that standardization through XML schemas is a means of making more efficient and hence less economically expensive. He states,

"One way to save resources on a long journey is to hitchhike. In XML, you can take advantage of countless open schema initiatives that, in effect, use schema standardization for top-down semantic transparency".

W3C (2004) has a number of XML Schema working groups. For any real detail in a concise format I would view Robin Cover's (2006) paper on XML schemas and the details on the various working groups and committees.

Uche Ogbuji (2005) "Thinking XML: Schema standardization for top-down semantic transparency" IBM (viewed on 17th July 2009)
W3C (2004) "XML Schema" W3C (viewed on 17th July 2009)
Robin Cover (2006) "XML Schemas", CoverPages (26 Nov 2006), Viewed 17 June 2009.

Exercise 11: Part 4

2009-06-17T09:34:00.002+10:00

Why are some developers using SAX instead of DOM for document processing?

In Clark (2001), Michel Rodriguez notes that;

"the vast majority of programmers that find DOM-type (tree-oriented) processing much easier to grasp than SAX processing. It feels much easier to "be in control" of the document and to act on it than to let it drive my code"

Like many others, this coder feels that SAX is overly complex. Being new and an additional learning exercise are a cause. However, it is not new in general, but new to the individual that counts in this statement.

However this is changing. The inclusion of an "event processing paradigm" is starting to make inroads for SAX. As more and more automated systems develop, SAX is likely to become more popular. This may be the future for Security and monitoring software...

Kendall Grant Clark (2001)"DOM and SAX Are Dead, Long Live DOM and SAX", XML.com, viewed from http://www.xml.com/lpt/a/872 on 17th June, 2009.

Exercise 11: Part 5

2009-06-17T09:19:00.001+10:00

SMIL is an application of XML. What is the purpose of this technology? Where does it apply?

Synchronized Multimedia Integration Language (or SMIL) defines an XML-like language that is designed to simplify the process used in publishing "interactive multimedia presentations". (Bulterman, 2008).

Through the integration of streaming audio, video, rich featured images, text and a multitude of other multi-media formats, the author is provided with a "rich media" canvas on which to create their digital works. (W3C, 2008).

Bulterman, B. (2008) "Synchronized multimedia integration language (smil 3.0)", W3C. viewed from http://www.w3.org/TR/2008/REC-SMIL3-20081201/ on 17th June, 2009.
W3C (2008) "Synchronized Multimedia", W3C. viewed from http://www.w3.org/AudioVideo/ on 17th June, 2009.

Exercise 11: Part 1

2009-06-16T22:14:00.001+10:00

Conduct research on the Internet to find out what tools can be used to parse an XML document and ensure that the document is well formed and valid.

A program that is used to validate the syntax of an XML document is known as a parser. There are of course parsers for many (most in fact) code languages. Parsing code is an integral part of the compilation process. In regards to XML, it is a process used to ensure that valid output will be created.

The reference page below [1] has a detailed list of XML/XSL Software Tools:

I still like UEStudio.

[1] Publicly Available Software for SGML/XML/DSSSL

Exercise 10: Part 11

2009-06-16T09:06:00.003+10:00

What are the differences between the various approaches to scripting?

The primary distinction in scripting is -

Client side
Server side

From a security perspective, client side scripting allows the user to interact and change the script. This is never a good idea. JavaScipt may be obscured by malicious code writers making the reversing of this information more difficult, but this is in an end to gain more control over the local host. The cost of this comes with the ability of the "good guys" (that is - us who reverse the malicious code) to analyse the scipt.

Exercise 10: Part 12

2009-06-16T09:05:00.001+10:00

List some of the shareware or freely available tools for web design. In addition to a good HTML editor, what utilities could be very handy to the developer?

The site, "WebSite Tips" is a good place to begin in answering this question.

A number of the good tools that are listed (and are free) include Arachnophilia, by P. Lutus. This tool is a:

HTML editor,
Programming editor
Text Editor
Web page development workshop, and
General programming tool.

With a combination of search & replace features, a spell checker, and an automated markup verifier (cleaner) you get a well featured tool for nothing (it even boasts an FTP tool).

It comes with native support for

ASP
HTML
XHTML
PHP, and
Java

My choice is still not free. My choice for Windows (UNIX is different again) is UltraEdit, by IDM Computer Solutions, Inc.

As has been stated, UltraEdit is a "reasonably priced text editor for Windows 98/Me/NT/2000/2003 and XP, and it supports 64-bit file handling (standard) on all 32-bit Windows platforms". It is my choice.

UltraEdit is the #1 selling, most powerful, value priced text editor available and is quite popular among programmers - “the ideal text, HEX, HTML, PHP, Java, Perl, Javascript, and programmer's editor!” From their site: “UltraEdit delivers easy viewing and editing of code and variables. In addition to the standard editing functions such as undo, copy and paste, and searching, UltraEdit's color-coded editor allows programmers to decipher different code types and variables, and you are going to love the powerful code-folding too! Additional productivity enhancing features include delimiter matching that shows you the location of the opening and closing of parentheses, and bookmarking which helps you remember and find significant sections of code such as subroutines and constant declarations.”

Exercise 10: Part 10

2009-06-16T09:04:00.001+10:00

What are the similarities and differences between ASP, JSP and PHP?

Of course, the initial difference is the language used. Each of ASP, JSP and PhP rely on separate languages. The code has deeper differences as well.

ASP is based on a propriety code base from Microsoft. This is based on the .Net active server pages. For the most part, this means C# or VB (Visual Basic). C++ and a few other languages are less common but available.

PhP is a Perl based scripting language that integrates well with MySQL. As an open source framework, the LAMP (Linux, Apache, MySQL, PhP) package has offered serious levels of competition to its better funded rivals in a way that has allowed many small organisations to gain access to e-commerce - though not always securely.

JSP is shorthand for Java Server pages and refers to a Java based framework that uses "Java Beans" in a manner reminiscent of .Net code. Like .Net, JSP is Object-Oriented, something that PhP can attempt to achieve through addons, but which is not quite there.

See the following links for more detail:
[1] Active Web Sites and Comparison of Scripting Languages
[2] jsp and microsoft asp
[3] What is the difference between JSP and PHP?

Exercise 10: Part 9

2009-06-16T09:03:00.002+10:00

What are the similarities between JavaScript, VBScript and ZOPE’s DTML?

To the average use, there should be little difference in whether JavaScript, VBScript or DTML has been used. The reality is of course a little different.

JavaScript (JS) and VBScript (VBS) are each interpreted languages and will often display differently on different browsers. In some cases they may also not be supported by the system. When a web page containing JS or VBS is downloaded, the complete script sourece is sent to the user. As a result, the code can be reversed or altered at the client (one principle of secure coding is to never trust client systems).

On the other hand, Zope has moded towards a server based language. DTML is implemented as a "server-side" language (really a script) that is designed to run on the server and just return the output to the client.

DTML commands are executed by Zope at the server. They are not sent to the client system. This means that the client can not intercept and change the code and it makes reverse engineering the site mroe difficult. As such, you recieve a dynamically constructed web page with fewer issues than JS or VBS.

Exercise 10: Part 8

2009-06-16T09:02:00.001+10:00

Why was XHTML developed? Do you think this addresses most of the HTML weaknesses?

W3C states that "XML™ is the shorthand name for Extensible Markup Language [XML]".

Simply put, XML was conceived as a way to create a markup language with the power of SGML with little of SGML's inherent complexity. Consequently, XHTML, was producted as a "reformulation of HTML 4 as an XML 1.0 application".

The W3C XML FAQ page details the majority of reasons why XHTML has been implemented and its multitude of benefits over simple HTML alone. Others (such as Paul McDonald, [1]) also take up this banner.

Others however disagree [2]. The main issue is that XHTML must be well formed. There is far less room for error in XHTML than with HTML.

Not all issues have been solved, but it is rare that this will ever occur. The efficiency certainly warrants the implementation of XHTML in many applications.

[1] Paul McDonald (2007) "The Case for XHTML" http://www.layouts4free.com/articles/view-article/49/3/0/
[2] Ian Hickson (2007) "Sending XHTML as text/html Considered Harmful" http://hixie.ch/advocacy/xhtml

Exercise 10: Part 7

2009-06-16T09:00:00.001+10:00

What are the differences between HTML and DHTML?

Simply put, DHTML (or Dynamic HTM) is an extension to HTML that incorporates the addition of basic animated items, dynamic menus and similar additions to a basic HTML Web page. In order to achieve this, DHTML code incorporates style sheets (CSS) and JavaScript to produce a richer web experience.

There is a downside. Javascript is not always available. Some older browsers and those with scripts disabled (or with plug-ins such as No-Script) will not benifit from the addition of these features. Many also do not use scripted languages from sites that they do not trust (most of the Web) as the one of the most common attacks has been via the browser of late.

Exercise 10: Part 6

2009-06-15T20:20:00.001+10:00

What is Customer Resource Management and why is it important to e-commerce?

CRM Today has a e-business e-zine (or online journal) focused on the topic of customer relations for one simple reason, business is about suppliers fulfilling the needs of their customers. In today's market, we have moved into a globalized economy. This has both opened the doors of opportunity and also exposed many firms to increased competition. As a consequence, it has become more crucial than ever for a supplier to ensure that it manages and retains its clients.

In hist article on client retention, Brown (1998) notes this point well. He notes:

"After the purchase, customers go through an information-seeking activity, seeking confirmation or acknowledgement that their purchase was intelligent. The more costly the purchase, the greater the customer's motivation in achieving the affirmation. There is enormous value in speaking with the customer after the purchase, serving to reassure them that they have purchased wisely and truly are getting their money's worth".

CRM is a means of achieving this end. This software (if used correctly) can aid the sales and marketing people in a firm in turning sales into returning clients. This is no less true in e-commerce than in the physical world. In-fact, it likely holds firmer with online clients being more fickle than bricks and mortar shoppers who get to know the individual the purchase from (which is demonstrated all too clearly in 's article [2]).

As Jeffrey F. Rayport (of Harvard Business School) notes in Griffith (1999) when looking at the Internet and a comparision to shopping malls
"When malls first started, they tended to be grouped around big department stores. As time went on, they increasingly broke down into specialist shops, because it was easier for consumers to find what they wanted that way. The question on the Internet is: 'Do these stores get smarter or dumber the more products they take on?" I think they get dumber."

CRM can be the solution to making e-commerce "smarter". The secret is not just have CRM, but using it wisely.

[1] Wendell J. Brown (Friday, June 19, 1998) "Retaining clients is as essential as getting new ones" The Business Review
[2] Richard Webb (2008) "Online shopping and the Harry Potter effect" (published 22 December 2008) Magazine issue 2687. New Scientist (http://www.newscientist.com)
[3] Victoria Griffith (1999) "Tailored Marketing on the Internet: Does It Really Capture Customers?" Strategy+business, Fourth Quarter, Booz&Co USA

Exercise 10: Part 5

2009-06-15T20:19:00.002+10:00

# Why is the perception getting stronger that integration will become a critical factor in coming days? What is the role of ERP within the enterprise software architecture?

For more than 15 years and in fields as synchronous to the Internet as publishing (Davenport, 1994) and as divergent as Benoit (2008) and Vogel/Barton(1998), authors have been noting the necessity of integrating systems.

ERP or Enterprise Resource Planning is a means of managing operational resources. As more and more systems become the norm in an organisation with firms packing virtualised hosts into racks full of servers, the needs to be able to manage these systems effectively and efficiently will only increase.

Add this to the existing but consistantly increasing need to improve all aspects of ones business by squeezing the last drops of efficency from all aspects of the organisation (from the warehouse to sales and all in between) and you start to get the reason for implementing an ERP.

It is about taking scarce resources and making them go further.

[1] Elisabeth Davenport (1994) "Perception of Economics in a Digital Publishing Environment: A Report of a Field Study" Interlending & Document Supply, Volume: 22, Issue: 4, Page: 8 - 16
MCB UP Ltd
[2] Jacques Benoit (2008) "Meeting IED Integration Cyber Security Challenges"; Cybectec Product and Technology Training Cooper Power Systems.
[3]Joe Vogel, & Bob Barton, (1998) "SCADA SYSTEM AT GOULBURN VALLEY WATER", 61st Annual Water Industry Engineers and Operators’ Conference Civic Centre - Shepparton
2 and 3 September, 1998

Go to the website of IBM, Oracle, Microsoft and Sybase

2009-06-09T15:52:00.004+10:00

In this exercise I have to:

Visit the website of IBM, Oracle, Microsoft and Sybase and search for any mention of e-commerce associated with their database products.
Next, review the suite or partnerships they list with related e-commerce offerings.
And finally compare their products with open source products such as MySQL.

Part 1
IBM has a wide range of products listed in this search. They tout WebSphere, Domino and DB/2 for a start. Next they talk of their expertise in Oracle, etc. They also upsel the Amazon Web E-Commerce Services.

Oracle push their database as a backend database through to a complete e-commerce solution.

Microsoft presents web service offerings, partner solutions, MS SQL, their B2B software and many e-commerce white papers.

Sybase has links to selected pages, but does not offer anywhere near the volume of material from the other sites.

Part 2
I am personally biased, MSSQL or Oracle - all based on the project and operation needs. Ignore ANYTHING on the websites, the BS factor is far too high.

Part 3
A comparison is simple, MySQL loses. It does have uses, but for my point of view, the capabilities of MySQL do not compare. I work with MSSQL and Oracle as they handle large databases. I have worked on databases of 68+ TB - MySQL does not handle anything close.

So what comparision...

Click to go back to exercise 10.

Zope

2009-06-09T14:31:00.001+10:00

In This question, we have to:

List some application servers in competition with ZOPE.
Access the web and find the latest version of the application servers mentioned in the text.
When were they released?
Can you make any conclusion about the release cycle of Internet software in comparison to other business software (e.g. spreadsheet, word processor, accounting packages)?

Zope is "Z Object Publishing Environment" which is a Python based object-oriented web application server that has come about as a part of the open source movement. Zope 2 is an amalgum of the following application frameworks into the Principia application server:

Bobo,
Document Template, and
BoboPOS .

At the time of writing, the versions are as follows with the dates and download links for Windows, Solaris, Linux :

Download Zope 2.11.3 (2009-05-02) the current Zope 2 stable release.
Download Zope 2.10.8 (2009-05-02) the current Zope 2 release from the previous stable branch.
Download Zope 3.4.0 (2009-01-29) the current Zope 3 stable release.

(See the following link for a further write-up on Zope).

In "The Zope Application Server Revisited, Installing and Configuring Zope" (Kevin Reichard, January 7, 2000), we see that Zope competes strongly where user access is required to be managed. This competes with WebDAV and IIS with the .Net framework directly, though ZOPE does integrate with IIS. Zope does not support COM and CORBA - a downside.

It is a publishing framework. It can be used as a means of uploading files (e.g. images) onto the web and simplifying file sharing. This also lends it's use to blog's and wiki's.

Zope differs from a standard web server in that it uses a database (instead of files). This is a standard Web 2.0 format. To this end it supports DTML (or Dynamic Text Markup Language) and Zope Page Templates (ZPT). ZPT is supposed to fiux many of the issues with DTML, though I have not investigated this personally.

Grok has developed as an enhanced framework for Zope.

As for the release cycle, it is difficult to make comparisions. This is an exercise in itself as a quantitiavive time series and classification exercise. I do plan to do something along these lines in the future, but not now.

Click to go back to exercise 10.

Rates for Web Hosting

2009-06-05T11:14:00.003+10:00

In my case, discovering the going rates for web site hosting is simple. I have a few domains hosted for companies I run.
It does vary a good deal based on what you are seeking from a hosting provider as to what costs you will face. As an example, in the two extremes, I have both an application hosting provision and a simple web server hosting provision.

The application hosting first off is with Zendesk. This is a cloud based web help desk engine. This costs $190 (US) each month. This is of course more than a simple website.

For hosting I use Whois. They have dedicated ecommerce servers using Linux and Windows with databases attached.

My costs per site average around $20 per month. This is for a large amount of storage, 20 MySQL databases, etc etc.

The costs start around $4US a month, but then the more you want, the more you pay.

Click to return to question 10.

The main web servers

2009-06-03T08:12:00.007+10:00

The main Web servers in use today and historically can be sourced using the Netcraft Web Server Survey.

As can be seen from the graph they produced (below), Apache still rules as the most common web server with Microsoft IIS second.

Vendor	Product	No. Sites	%
Apache	Apache	96,531,033	52.05%
Microsoft	IIS	61,023,474	32.90%
Google	GWS	9,864,303	5.32%
nginx	nginx	3,462,551	1.87%
lighttpd	lighttpd	2,989,416	1.61%
Oversee	Oversee	1,847,039	1.00%
Others	-	9,756,650	5.26%
Total	-	185,474,466	100.00%

From the Netcraft table above we see the Rusian nginx server has made a small inroad into the total number of installed systems.

From the graph below we see that Apache has peaked and is losing ground to newer competitors. Microsoft IIS has a level of stability around 25-30% of the market.

Many newer servers have been appearing. These are reducing the share of Apache servers. These include:

Ruby's application server Mongrel,
Zope Pike (a Python based web server),
Caudium,
The Erlang based Yaws, and
The Haskell coded Salvia.

As a consequence, it is easy to see that Microsoft and the Apache foundation are leading the web server war. Google (with just over 5%) has made significant progress in this area as a late comer, but has a long way to go.

Many "appliance" vendors are using Apache. Add to this the Linux and Unix market as well as other related or similar frameworks that offer Apache server by default, and it seems unlikely that Apache will lose the lead in the short term.

Apache is also open source and free. IIS comes freely with Windows Server, but this adds an in initial cost through the requirement to buy the OS. This makes the use of IIS as an appliance less likely.

Support for IIS is provided from Microsoft and a number of consulting firms. Apache is more widespread and less centralised. Although the Apache foundation releases the source code, the support is provided by a dispersed assortment of companies such as:

Red Hat
HP
Consultants, etc

The Web server purchase price may or may not be a significant factor in budget considerations for a large e-commerce initiative. This all depends on the initiative. Appliance manufactorers are likely to be impacted as the OS and hence web server costs can be a significant addition to their cost, but in a hosting company environment where server thousand servers can reside on the same system, the cost of the OS diminishes.

The functionality and skillsets of individuals comes to be more of a deciding factor. For instance, if you have .Net developers you are likely to deploy IIS whereas a Ruby developer may go for any number of platforms from Apache to Mongrel.

Back to Question 10.

Exercise 9 - PHP Code

2009-06-03T08:07:00.001+10:00

I have listed the PHP code used for exercise 9 below:

< ;?php
echo "< ;HTML>";
echo "< ;HEAD>< ;/HEAD>";
echo "< ;BODY>";
echo "< ;b>This is the output to Exercise 9< ;/b>";
echo "< ;BR>";
echo "< ;BR>";

if($card == "American") {
$denum = "American Express";
} elseif($card == "Master") {
$denum = "Master Card";
} elseif($card == "Visa") {
$denum = "Visa";
}

echo "< ;BR>";
echo "< ;b>Hi there< ;/B>, < ;i>", $name;
echo ".< ;/i>< ;br>< ;br> You have selected to buy< ;b> ", $order;
echo "< ;/font>< ;/b> from < ;i>< ;b>ça sent le< ;/B>< ;/i>< ;br>< ;br>< ;br>";
echo "You are connected from a server with IP Address: < ;b>",$REMOTE_ADDR;
echo "< ;BR>< ;br>< ;/b>";
echo "< ;br>< ;blink>YOU ARE DAFT!!!< ;/blink>< ;br>";
echo "< ;br>< ;br>";
echo "< ;blink>Beginning to drain funds to illegal overseas account...< ;/blink>< ;br>< ;br>< ;br>< ;br>";
echo "Processing ", $name;
echo "'s ", $card;
echo " card with number... ", $number;
echo "< ;br>< ;br>";

if($card == "American") {
$pattern = "/^([34|37]{2})([0-9]{13})$/";//American Express
if (preg_match($pattern,$number)) {
$verified = true;
} else {
$verified = false;
}

} elseif($card == "Dinners") {
$pattern = "/^([30|36|38]{2})([0-9]{12})$/";//Diner's Club
if (preg_match($pattern,$number)) {
$verified = true;
} else {
$verified = false;
}

} elseif($card == "Master") {
$pattern = "/^([51|52|53|54|55]{2})([0-9]{14})$/";//Mastercard
if (preg_match($pattern,$number)) {
$verified = true;
} else {
$verified = false;
}

} elseif($card == "Visa") {
$pattern = "/^([4]{1})([0-9]{12,15})$/";//Visa
if (preg_match($pattern,$number)) {
$verified = true;
} else {
$verified = false;
}

}

if($verified == false) {
//Do something here in case the validation fails
echo "Credit card invalid. Please make sure that you entered a valid < ;em>" . $denum . "< ;/em> credit card ";

} else { //if it will pass...do something
echo "Your < ;em>" . $denum . "< ;/em> credit card is valid.< ;br>";
echo "We have extracted ", rand(1000, 100000);
echo " dollars from your ", $denum;
echo ".< ;br>< ;br>";
echo "Your cat's piss... Sorry, ", $order;
echo " will be delivered within 20 years. Promise!< ;br>";
echo "< ;br>";
echo "< ;br>";
}

?>

The rest was simple HTML.

Click here to go back to Exercise 9.

My Development Experience

2009-05-26T21:45:00.002+10:00

My Development Experience has occurred over a large number of years and I do not remember half of the range of languages I have covered. I shall list a number of the programming languages and Web development tools I have used in prior experiences, but this is FAR from the totality of them.

Programming Languages I have experience with:

Intel Assembly
Sparc Assembly
C
Fortran
R (a statistical Programming language)
S-plus
SAS
C ++
C #
.Net (1.0, 2.0 3.5)
SQL (Oracle PL-SQL, T-SQL etc)
Fred
CGI (in a variety of ways)
TCL
SED/AWK
Many Shell Languages
DB2
PIC
SPICE
BASIC (ARG!)
PASCAL (ARGGGGGG!!!!)
javascript
Flash
PHP
Perl

There are more, but these are all I can remember for now.

I have worked with:

XML
XBRL
HTML
UNIX Sockets
SQL Server - Stored procedures & DTS

On top of this I do extensive work with regards to code reversing and malware analysis these days.

The Web application frameworks I use are simple for the most part:

MS Visual Studio.Net
VI (Unix Text Editor)
Eclipse (Java and C)
Matlab (it has a web engine)

And I am finally getting into Ruby.

I mostly conduct code analysis (looking for security flaws in source code) and write reversed code functions (e.g. for cryptanalysis and in Malware analysis). One of my job functions is to create graph diagrams, psuedo-code, C Code and CFGs (context free grammers) for packers (such as Themida).

Assessment item 1

2009-05-26T12:42:00.004+10:00

This page links to the various sub-pages and posts associated with Assignment No.1.

The first Assignment comprises of 11 exercises:

Ruby on Rails Workshops 1 to 4

And the first Elevator Pitch.

Elevator pitch 1 on progress report for assessment item 1

Exercise 8 - The answers

2009-05-26T09:19:00.003+10:00

Click Here to go back to the Exercise.

And Here for the answer...

When in Windows...

2009-05-25T22:28:00.002+10:00

When working in a Windows environment (UNIX being "vi"), I have a definite preference for Ultra Studio. This IDE is combined with Ultra Edit.
This is a great Hex/Text and script editor. I have their entire software range. A great addition of any Windows based coding environment.

A plug and I am not even getting paid...

Computing at the speed of Ruby

2009-05-21T21:49:00.002+10:00

With this subject (ITC565) I have been working with and learning Ruby (and for that matter rails). I do a fair amount of algorithmic coding. This is mathematical and statistical (and cryptographic) coding. This requires each and every cycle.

With Ruby you lose cycles. The argument is that you do not require speed for web based systems, but I have generally found this to be untrue for the clients I have worked with. Even companies such as eBay sacrifice security (by removing many of the SSL encapsulated pages and sending in clear text) for the sake of cycles.

There are some beneficial aspects of Ruby. I do see this, but it is slow. Coming from a C background and foundation, I like the speed and simplicity of C (a type of purity).

The nature of Ruby (in being an interpreted language and not compiled) has both benefits. In reversing code, it makes life a good deal simpler. For this reason alone I do not see a good deal of malicious code being based on Ruby. It is far too simple and easy to reverse and analyse (one of my tasks).

Now, if I can just find a way to have malware writers use Ruby...

Workshop 1 - Setting up the model railway

2009-05-21T10:18:00.005+10:00

Topic objectives
The first workshop is designed to

Install Ruby on Rails on your computer (InstantRails or Locomotive);
Learn about the Model View Controller (MVC) approach to Web application design;
Revise database techniques with MySQL
Learn how to use the Ruby on Rails development environment
Set up a focus group (like a study group for peer learning) to work on the Ruby on Rails workshops via Interact tools

Topic readings
Various websites as listed.

Discussion
One way to learn a Web application framework is by building a project. It’s like driving a train – you can’t learn to do it by reading a book. In this workshop series we shall use the Online Taxi Booking System as the “train”, you the developers are the passengers, and the train rides the Rails under commands from Ruby.

The Project
The sample Ruby on Rails Project is the Online Taxi Booking System as briefly described in the subject outline with more details added later during session via the forum or CSU Interact. Use database design techniques and the model view controller design (MVC) approach. The database-driven website is developed and tested on your local PC using Ruby on rails and Web browser, before being migrated to a production site on the CSU network.

The database called taxi has TWO tables called passenger_origin and passenger_destination:

Passenger_origin:
Name, Contact No., Suburb, Street, Street No., Building (Unit, house or business);

Passenger_destination:
Suburb, No. of passengers (1 to 4), taxi type, time required;

To do:

Download iTunes from http://www.apple.com/itunes/download/ and subscribe to the “Leraning Rails” Podcasts from http://www.buildingwebapps.com/podcasts
Install Ruby on Rails on your computer by using the material and downloads from http://www.rubyonrails.org/
Rather than get Ruby on Rails running manually, you use the pre-packaged solutions. These include everything in one bundle: Web server, database, Ruby, Rails, the works.

a. For OS X, there's Locomotive.
b. For Windows, there's Instant Rails.
Nothing listed for Linux in the workshop list...

I have selected a different option. Though I have started with a On Click Installer solution to get an idea for Ruby, I am going to go from the start and compile Ruby on Windows. I have a database and webserver ready, so I as usual will try the hard way. I find I learn more this way (if you ignore the frequent swearing). I do recommend that you go the pre-compiled option. The integration of Visual Studio 2008 and Cygwin is a slow and error prone process.

The install of the pre-packaged version was completed in under 20 minutes. Compiling Ruby took over 9 hours and I still have a few bugs to iron out.

Challenge Problems:

Make a list of all programming languages and Web development tools used by you in prior experiences. Describe what you know about Web application frameworks before we begin.
Ruby is “an interpreted scripting language” for quick and easy object-oriented programming”. Find out about the Ruby language and discover what this means.
What is Rails and how does it work with Ruby?
What is meant by “convention over configuration” in regards to the use of Rails in Web application development?
When did Model-View-Controller begin and where is it used?
Describe the steps involved with the MVC design approach.

Well I am signed up for the audio lessons at least:

Compiling Ruby on Windows

2009-05-21T09:57:00.006+10:00

I have downloaded the source for Ruby (ruby-1.9.1-p129.zip) from the same site where I obtained the "One Click Installer".

On downloading the initial stage involves extracting the source files:
This process requires cygwin to be installed. I learnt this when attempting to compile in Visual Studio 2008:

I will not go into the install and compile process here. Rather, I will recommend that you go to Rails site - the http://rubyforge.org/frs/?group_id=904&release_id=17517 and use the complete package.

It was a learning experience in compiling Ruby from source, but it is not one that I will repeat.

MySQL and Apache are far simpler for this. Additionally, I found that compiling Ruby (and the other software) in Windows was far more problematic than under Linux (in this case SuSE and Redhat). The issue is Cygwin. There is a lot of work required to have Visual Studio 2008 and Cygwin talk nicely.

Ah well, live and learn.

Running Ruby

2009-05-21T08:35:00.009+10:00

Last night I installed and configured Ruby for my system using the "One Click Install" process. I must admit, they have made this increadibly simple. So now to running it. I have my copy stored and accessed under a path that I know (creature of habit).

From this menu I have selected first the SciTE editor function/compiler/etc. The Language menu demonstrates a good range of supported languages as you can clearly see from the dropdown menu below. Far more than Ruby.
So to start I have loaded a sockets based sample program designed to connect to a remote system.

Enteriong "F5" or Go runs the program. This is basically a debugging operation. I did this a few times (as displayed below) chaning the hosts and ports to see how this worked.
I eventually connected to an SSH server on TCP 22 at work - though with no crypto logic built in, the communication was sparse. The server (SSH) version was returned.

Next, I opened the IRB (Interactive Ruby Console).

Based on the examples in the book and on the sample code, I created a small sockets based program to conntect to the SSH server again. As you can see from the diagram above, this connected and the server returned the version of SSH running on this system, "SSH-2.0-OpenSSH_4.3".

Of course, directly after this, the system returned a Protocol Mismatch error. This was expected. I have only made a simple clear text connection to the SSH server. My code does not include any of the required protocols for talking to an SSH server as yet. It just sets up the initial handshake and recieves the SSH version from the server.

From here I could progress a number of ways. For instance, I could use the version information to determine if my app could "talk" with the SSH server (i.e. I may or may not include code for SSH 1.x) and drop unsupported protocols.

Or as I shall do, I could write the version to a database with the host information. This then gives me a very simple SSH version scanner in a few minutes of coding. To make this useful, I would have to save the output (at present it is just displaying on screen). The reason is that I could then walk away. If the process crashed, I can come back and continue from the last tested system (or IP).

To do this, I would add a simple for, next loop. Add a start and finish IP address and scan from one to the other. IP addresses can be displayed in a number of ways. For instance, the IP address 203.57.21.5 used before can be displayed as the value, 3,409,515,781.

This is calculated as:
(203 * 256 * 256 * 256) + (57 * 256 * 256) + (21 * 256) + 5 = 3 409 515 781

As a consequence, scanning the values, 3,409,515,777 to 3,409,515,786 is the same as scanning the IP range: 203.57.21.1 to 203.57.21.10.

Over the coming weeks I shall expand this project and include code in Ruby that acts as a simple SSH version scanner.

Installing Ruby

2009-05-20T22:30:00.010+10:00

Here I am continuing with the install of Ruby. Being a C/C++ and Assembly dinosaur, I have not touched Ruby before today. Then again, I reverse malware, write anti-packer software etc. None of this has any relation to what Ruby does. However, we all need to learn new things.

The initial stage is the acceptance of the license.
I note the use of Visual C++ in the license, so l.ater in the week when I load this into Microsoft VC 2008 and compile the source, I hope to have few issues. You need to accept the code - or you get no further. I so far understand the source of the compiler better than the "easy" 4GL that is Ruby.

I have (see above) selected "Ruby Gems" as an additional option to install as these are mentioned in a couple of papers and books on Ruby that I have [Ullman, 2009). I have not used these, but some of the examples seem interesting in the books.

The default install location is displayed next.
This is "C:\Ruby". I standardise my Windsows systems using "C:\Programming" to hold most of the languages and comilers I have under a common directory (this is just me and I find this easier).

As a result, I have selected the directory displayed below:
The same goes for my Sub Menues. I will install these under a path I seledct where possible. There are a couple of these to be updated. "Open Office" has set its own on an update (though it was correct). The same with a couple others.
Now that Ruby is selected under the "Start Menu" "Programming Tools" folder, it is time to install it. I have selected "install" and as you can see, it is running.
And it not only ran, but it installed correctly first time. This is positive.
Once completed you have a finalization screen and (if selected) you can read the Readme. I generally think this a good thing.

See:
Ullman, Larry (2009) "Ruby, Visual Quickstart Guide" Peachpit Press

Blog Linking

2009-05-20T20:38:00.003+10:00

Grant asked about why my page did not have a "links to this post", section on my blog.

This feature is known as "Back Linking". Google - Blogger have an FAQ on this topic.

In my case, I am not using this as I can search in Google for linked pages easily (manually). As such I do not use it.

To configure this feature, you have to go to your Blogger configuration. From here, select the "settings" tab as shown in the image below:
Scrolling down, you come to a section on Backlinking. As you can see from the image below, I have this set to "Hide":

Google Searching for links to your pages
Most (in fact all and more) of the above functionality can be obtained from the use of a Google search. In this case, we need the "link:" command.

The “link:” command syntax is used to list those webpages with hyperlinks to a specified webpage.

In my case;
“link:information-defense.com” is used in order to display the list of webpages containing hyperlinks to my companies website (http://www.information-defense.com) and homepage in this example.

Personally, I do NOT place a good deal of faith in this command (or that of bloggers backlink function). They do NOT work well at all.

If we take the search, "link:sansforensics.wordpress.com" the return rate is between 240 and 250 links at the time of posting. The reality is that I have no idea how many pages link to this, just that I could find over 1,000 that do. Hence the rate of success for this command is dismally low.

Note: It is essential that there are no white-spaces entered between the "link:" and the web page URL following the command. This will be treated as a separate item if the whitespace is included.

See:
Long, Johny (2005), "Google Hacking for Penetration Testers" Syngress

p0wf (Passing Fingerprinting of Web Content Frameworks)

2009-05-20T08:30:00.000+10:00

“p0wf” stands for “Passing Fingerprinting of Web Content Frameworks”.

Traditional OS fingerprinting used the OS Kernel to identify a system it is communicating with. This was based on the idea that if one can identify the kernel, one can target daemons that tend to be associated with it. The web has become almost an entirely separate OS layer of its own, and especially with AJAX and Web 2.0, new forms of RPC and marshaling are showing up faster than anyone can identify.

p0wf was designed to analyze these streams and determine just which frameworks are being exposed on what sites from the traffic alone. The primary distinction with p0wf is that it can analyze “sniffed” traffic and not alert the site being monitored of its presence.

(Material in this post has been extracted from some of my books and other papers).