Refresh Software

Bolero 2, Everyware Development Inc.’s new Web server analysis tool, gives site administrators good reporting capabilities in real time, as long as their Web server is one of the few supported by Bolero.

Nevertheless, in recent tests, we were impressed with the reporting capabilities in Bolero 2, which shipped last month for $13,500. In addition, once we got past a few setup glitches, we found the management interface simple and intuitive.

Bolero’s price is comparable to that of Andromedia Corp.’s competing product, Aria, and other high-end log analysis tools.

The Bolero 2 package includes the Bolero Server, the reporting system, a SQL database and one Server Agent. (Additional Agents must be purchased separately.) Also included is a limited-license version of Everyware’s popular Tango Web development application for report customization. Many of Bolero’s reporting features are based on Tango.

However, many companies will be unable to take advantage of these features, even organizations that use the freeware Apache, which is the most commonly used Web server, because Bolero works only with servers from Microsoft Corp. and Netscape Communications Corp. It doesn’t even work with the latest server release from either company.

Alternatives include WindDance Networks Corp.’s WebChallenger and other products that use packet-sniffing technology, which is less detailed than other technologies but works with any Web server.

Plug in, turn on

Bolero 2 uses a Web server plug-in as an agent to provide activity reports in real time. This approach can free administrators from having to manage the extremely large access logs used by most analysis tools.

Bolero’s approach is not unique: Aria 2.0 also uses a server agent to provide real-time analysis. Bolero 2 has the same main flaw found in the original Aria 1.0: very limited Web server support.

Setting up Bolero was a little touchy in tests. We found that the agent wouldn’t work properly if we didn’t stop the Web server before installing the package, and we ran into some problems setting up security access. However, these were relatively minor problems, and everything went smoothly once we solved them.

Before we could begin analyzing site activity, we first had to configure the server and the agent. All of this is managed centrally via a browser, and a wizard program takes site administrators through the process step-by-step.

Just the (relevant) facts

Bolero provides site administrators with plenty of options for filtering out unimportant data, so only relevant information is stored in the database. We were impressed by the wide variety of filtering options, and we could also choose to filter content at the agent or at the Bolero server.

We were able to import current log data into the Bolero database using a command-line program. Although this program was simple and straightforward, we would like to see this functionality built into the main management interface. After this, we were able to schedule regular log imports, which allows companies to use both real-time analysis and standard log analysis.

Once archived data was imported, we started to capture live user activity. Busy sites will appreciate Bolero’s ability to provide detailed real-time reports that can be easily accessed via a Web browser. However, Bolero can output reports only as HTML. Although this is fine for most uses, many companies would no doubt like to be able to output reports to word processors as well.

The bundled reports cover most of the information that a site administrator needs, such as visitor and referrer identification and cost analysis. Those who are unfamiliar with Tango might be a little intimidated at first, but the bundled Tango editor allows businesses to create very thorough reports.

Bolero 2 filters out data and can block search agents’ activities.

For businesses that want detailed analysis of activity on their Web sites and want it in real time, Everyware’s Bolero is a good choice, providing strong report customization capabilities. However, the product supports only a small number of Web servers, meaning many companies will have to look to other tools for real-time analysis.

Your server can generate two or more log files with information about the transactions it completes. The most important information from a management standpoint is contained in the access log and the error log. If you want to know what’s happening on your site, look at your access log, which contains information about every completed HTTP transaction.

An access log file shows the visitor’s IP address, the date and time a page is requested, the name and location of the file requested, a status code, and the number of bytes transferred. Unfortunately, unless your site includes only a handful of simple pages and gets few visitors, simply browsing through the information in your access logs rapidly becomes impossible. A site with even a moderate volume of visitors and a few dozen pages will likely feature access logs with thousands of entries per week, rendering direct viewing of the entries impractical.

You can harness the power of your access logs by using software to analyze the data they contain. Most commercial Web servers, such as Microsoft’s Site Server and Netscape Communications’ Enterprise Server, include log file analysis tools.

There are also dozens of commercial log file analysis tools available at a relatively low price, such as WebTrends’ (Portland, OR) WebTrends ($299) and WebManage Technologies’ (Nashua, NH) NetIntellect ($199). There are also numerous freeware options that provide equally detailed reporting, albeit without the extensive documentation and support of the commercial products. One freeware example is wwwstat by Roy Fielding of the University of California, Irvine. A Perl script written for Unix-based systems, it generates detailed, table-based traffic reports in HTML format and integrates with gwstat another freeware program to graphically display site traffic.

Log file analyzers operate on the same principle: They parse each line in the access log, populate a database with the parsed data, and build reports based on a variety of queries. The most basic packages provide information such as total number of hits, impressions (pages viewed), least frequently and most frequently visited pages, number of kilobytes transferred, and number of client or server errors.

Most analyzers further report these measurements in daily, hourly, or even shorter time increments. Such finite detail provides valuable information about overall traffic patterns, peak traffic periods, data transfer rates, and problem files on your site. Table 1 (page 38), generated by WebTrends, shows average daily traffic, peak traffic periods, and low traffic periods.

Because of the stateless nature of Web transactions, log file analyzers necessarily make some assumptions about the data pulled from the access logs. For example, there might be a fixed (or possibly configurable) time span that is used to determine a session time-out. The log file analyzer identifies sequential page requests from one IP address as a single user session, and assumes a session time-out when it encounters a time gap greater than the fixed time span. In this way, the software can deliver relatively accurate information about the total number of user sessions (visits to the site), average session lengths, common paths through the site, and more.

CRITICAL QUESTIONS

Beyond the basic reporting options described previously, log file analyzers offer a variety of advanced features, many of which are not only desirable but possibly crucial to keeping your environment running smoothly. You need to take your site’s configuration into account, as well as any advanced reporting you and your users will require if you’re planning to add a log file analyzer to your site, or upgrade your existing software.

Ask yourself the following questions about your log file analyzer:

Can it read and synthesize extended log file formats? Most commercial servers let you capture extended information in your access logs, such as browser type and version. Make sure you know what options your server offers for capturing access statistics, and verify that the analysis software supports those formats as well.

Can it merge logs from multiple servers? If you are load balancing multiple servers, or need to merge traffic information from multiple sites, this option lets you measure traffic across all servers on the site.

What methods does it support for accessing access log files? For example, does it support direct file access, HTTP, FTP, or some combination of the three? Also, can it support name and password authentication for proxy or remote file access?

Does it include a scheduler function for prearranging future reports? If you need to generate periodic reports for one or more sites, report scheduling helps reduce your workload.

Does it support real-time reporting of log file data? If it’s important for you to have up-to-the-minute information about your site’s traffic, this feature is a must.

How customizable are the reports it generates? You won’t find a one- size-fits-all solution in any of the log file analysis packages on the market. The more flexibility you have to configure reports and filter report data by factors such as time, file type, and directory, the better. If you need heavy-duty reporting capabilities, look for a tool that can export the compiled data to an external database, allowing you to create customized reports and queries.

A typical server log file.

Other possible options to look for include automatic conversion of IP addresses to domain names, report delivery via e-mail, and the capability to output reports in non-HTML text formats, such as spreadsheet or word-processor formats.

In addition to the access log, your server also generates an error log file, which is a record of failed HTTP transactions such as unauthorized access or missing file errors.

It’s good practice to browse your error log on a periodic basis to find problems and security breaches on your site. Consecutive failed attempts to access secure areas of your site may indicate someone trying to exploit a security hole on the site. You can also quickly identify broken links, missing pages, and misplaced image files.

While there aren’t tools available to analyze error logs, the error logs are generally not as large as the access logs, and thus are easier to browse with a text editor.

CORRALLING YOUR CONTENT

Anyone involved with creating or maintaining content on a Web site understands how difficult it is to control the integrity of the site’s resources. Problems such as broken links, missing files, or poorly coded HTML crop up all too often in environments featuring multiple authors and internal and external resources that are in a constant state of flux.

Even with tight controls on authoring and administration practices, it’s inevitable that there will be structural problems scattered throughout a site. If this scenario describes your Web environment, it’s likely your production and development personnel devote a significant amount of time to locating and correcting such problems.

Help is available in Web content management tools, which help root out and correct problem files. Like log file analysis tools, most products are competitively priced, such as Tetranet Software’s (Kanata, Ontario) Linkbot ($249) and Site Technologies’ (Scotts Valley, CA) SiteSweeper Workstation ($295).

These tools, also known as link checkers or site mappers, provide quality control by sifting through a site in search of broken links, missing images, poorly constructed HTML code, and other problems. Most can display a visual roadmap of a site, making it easier to understand the site’s structure and find files or sections quickly. Figure 1 shows an example of a site map as implemented by Mercury Interactive‘s (Sunnyvale, CA) Astra SiteManager.

After directing the link checker to a site’s home URL, it retrieves the index page and searches for hypertext links, image tags, and other media links embedded within the code. It tests all the resources, verifying that they exist and load properly. The link checker follows the same procedure for each page linked to the index page, collecting information about the page’s resources and following any hypertext links it encounters. In this manner, it continues to sift through a site, building a catalog of information about all the files it processes.

Content management tools may offer a wide variety of functions, including testing for broken links, building a visual map of the site, locating orphans (unused files no longer linked to any pages in the Web site), identifying slow-loading pages, and possibly locating and repairing incorrectly coded HTML.

The available tools fall into two basic categories: products offering better mapping and navigational abilities, and products better suited to reporting and correcting errors.

THE RIGHT TOOL

As with log file analysis tools, you should know what features are needed for your environment and select the tool accordingly. If you’re thinking about adding content management software to your site, consider the following questions:

How does it help find and repair broken links? All these tools should help root out and repair broken links, but how they go about it differs. If broken links are the biggest problem on your site, look for tools that offer advanced error-reporting capabilities rather than intuitive visual mapping. Also, if you need to correct numerous errors on a regular basis, check to see that the tool can integrate with your third-party editing software, or that it includes built-in editing software.

Can it locate outdated files? If your site’s resources change or rotate frequently, look for a tool that can test and report the last time files were saved.

Can it test for slow-loading pages? If you’re concerned about alienating bandwidth-challenged visitors, this helps identify excessively large files that might result in aborted transactions. Some tools can help locate problem files based on your minimum bandwidth standard say, for a 28.8Kbit/sec or faster modem.

What errors can it detect in HTML code? This feature could help isolate and correct inconsistencies resulting from poor coding techniques. If this is a common problem on your site, you should find out to what extent the tool analyzes HTML files at the code level. Some common problems are duplicated title tags, incomplete or missing head information, and images with missing alt, height, or width tags.

Does it support selective scanning? Filtering by directory, file type, or number of link levels allows you to focus on a subset of your site, which might help reduce the time the tool needs to read and analyze the site.

How does it display a site’s structure? If your greatest need is for a tool that helps you visualize your site’s overall structure and organization, look for one with more extensive and intuitive mapping features.

How will the product scale as your Web site grows? If your site is already large or growing rapidly, you need a product that won’t get bogged down by thousands of documents. While vendors might claim that their products are scalable, you should test several such products on your site before making a purchase.

Can it support form-generated CGI content? Some tools let you preset variables to be entered on your site’s CGI forms, enabling you to test and map dynamically generated pages.

Overall, content management software products are still relatively immature but are evolving quickly; look for rapid upgrades and expanding feature sets in the coming year.

Although none of the currently available tools provides excellent support for all the requirements and functions we’ve discussed in this article, Webmasters trying to maintain large or frequently changing sites can benefit from the visualization and error-detection capabilities these tools provide. Most vendors offer downloadable evaluation versions of their products on the Web, so you’d be wise to try out several to find the one that best matches your site’s content management needs.

NOW IS THE TIME

The number of Web management tools on the market has mushroomed within a relatively short time span. Competition is rampant, with vendors scrambling to expand their product lines to cover a wider range of management functions. This is good news for the consumer, who can expect more innovations and tools that integrate a growing number of solutions.

Although this trend should continue for a least another year or two, that doesn’t mean you should wait to get help managing your Web site. Most of the products available provide a wealth of information and utility at very affordable prices.

As long as you understand what these tools really offer, their limitations, and how well they integrate into your environment, you can reap substantial benefits from them today.

Phil Keppeler, Network Magazine’s Webmaster, can be reached at pkeppele@@mfi.com.

There’s a diverse range of products whose makers claim are Web management tools. If you’re planning to expand your site’s Web management capabilities, you need to understand what each type of tool manages, and whether it applies to the needs of your environment.

The phrase Web site management tools describes anything from basic authoring tools to large, enterprise-level Web development and deployment systems. A breakdown follows of how the market is currently categorized.

Integrated Web authoring tools. Examples are Haht Software’s (Raleigh, NC) Haht Site and NetObjects’ (Redwood City, CA) Fusion. In addition to providing a wide range of design and development functions, they help Webmasters manage site structure through site mapping and visualizing features.

Traffic analysis tools. These include products such as WebTrends’ WebTrends (Portland, OR) and net.Genesis’ (Cambridge, MA) net.Analysis Pro, which slice and dice a server’s log files for comprehensive site traffic and usage reporting.

Link checkers and site mappers. This category includes Mercury Interactive’s (Sunnyvale, CA) Astra SiteManager and Site Technologies’ (Scotts Valley, CA) Site Sweeper. These tools sift through a server’s content to create a map of a site’s structure and report on structural problems, such as broken links, within files.

Performance monitors. Examples are Avesta’s (Nepean, Ontario) Webwatcher and Network Associates’ (Santa Clara, CA) WebSniffer fall into this category. These utilities monitor the availability and performance of network resources. In addition to providing feedback about your network’s performance, they automatically alert you when network services fall outside of an acceptable range.

Bandwidth managers. Examples include RND Network’s (Mahwah, NJ) hardware-based Web Server Director and Resonate’s (Mountain View, CA) software-based Dispatch. These products can help those responsible for the network infrastructure to manage bandwidth and load over network resources. There are many hardware, software, and hardware/software combination solutions that distribute Web traffic among multiple servers in a local or distributed environment. Some help manage servers in such environments, and some help manage the content resources throughout the Web environment.

Workgroup management and version control tools. Two offerings in this category are MKS’ (Waterloo, Ontario) Web Integrity and Wallop Software’s (Foster City, CA) BuildIT. They provide file check-in and check-out, as well as version control for complex sites with a workgroup development environment.

Comprehensive enterprise development systems. Example offerings are Vignette’s (Austin, TX) StoryServer and Inso’s (Boston) DynaBase. Products in this category provide a complete framework for producing complex, data-driven sites that may include connections to back-end data sources, workgroup development environments, e-commerce, and more.

NetObjects Fusion was one of my favorite HTML editors, at least back in the days pre-CMS. Check this review out:

The program is divided into five views: Site, Page, Style, Assets, and Publish. The Site view reduces your site’s pages to a series of connected icons, making it easy to view and rearrange the parent-child structure of your site. To move a page, you simply drag and drop it to a new location–a real time-saver for growing Web sites. Navigation tools, links between pages, banner titles, and other page-specific elements are automatically updated when you create or move a page. This internal linking of elements makes it easy to try out different page groupings. New with version 3.0 is the ability to select some of the pages in the Site view and apply functions to them as a group.

The Page view is the area that most resembles a desktop publishing program. Tool palettes let you add text, graphics, forms, and tables, as well as shapes, multimedia content, and third-party add-ins (including Java applets, Shockwave files, and ActiveX files).

You can also add NetObjects Fusion Components. Components are small JavaScript programs that add specialized functions to your Web pages. Nine Components ship with the program, including a message board and a site map. Others are available from third-party developers.

Dynamic HTML is handled through Fusion’s Dynamic Actions. These let you apply complex animations and interactivity to objects by selecting options from a menu box. While there are differences in how Netscape and Microsoft have implemented Dynamic HTML in their version 4.0 browsers, Fusion gets around these differences by having its Dynamic Actions use a combination of Cascading Style Sheet (CSS) positioning, JavaScript, and browser-specific HTML code.

Along with the traditional graphical page view, version 3.0 provides two other page views. The textual page view is for pages that are primarily text. It writes leaner code than the graphical page view does, so its pages load faster. The external HTML page view is designed to incorporate existing HTML pages–or portions of pages–without modifying the original HTML code. This is necessary because otherwise, if you already have HTML pages, Fusion alters the code to such a degree that most programmers are unable to work with it. And if you make any changes to existing code, the program will automatically translate those changes as well.

If it’s essential that you be able to modify your code, you may be better off with Macromedia’s Dreamweaver, which never alters code it doesn’t recognize as standard. Microsoft’s FrontPage 98, on the other hand, always edits the code to suit its own conventions. Fusion takes a middle course and lets you mix modified and unmodified pages.

The Style view lets you switch collections of design elements, such as banners, navigation buttons, backgrounds, and text colors. You can move from one style to another with a few mouse clicks, altering the entire look and feel of a site. Novices may find this one of Fusion’s most useful features. But it’s also a powerful tool for experienced Web designers, especially when the styles are combined with the MasterBorders feature, which automatically repeats elements across multiple pages. Using both styles and MasterBorders, you can add new pages in a matter of seconds and have each new page share design elements with all your other pages.

Unless your site is unusually large, you probably won’t spend a lot of time in the Assets view. This view lets you update or reassign a resource, such as an external file, link, or data object, and have that change automatically reflected on every page that uses the resource. The Publish view, new to version 3.0, lets you choose a directory structure for the site’s files. The structure can be organized by asset type (handy if you’re using a large number of graphics or database files), by site section (useful if your site has a logical structure), or in a flat mode (necessary for servers that require all files to be in a single directory). The Publish view provides three HTML output options: Nested Tables, Regular Tables, and CSS and Layers.

Fusion generates code that can be interpreted–or selectively ignored–by 2.x, 3.x, and 4.x browsers. The company refers to this process as Everywhere HTML. But while the results are impressive, they aren’t perfect. For example, we were able to create a moving graphic using Dynamic Actions that locked up a Windows CE browser whenever the page was loaded. Generally, the further you move away from 2.x-level HTML, the more testing and tweaking you’ll need to do. The same is true for other authoring software as well. Until Netscape and Microsoft can agree on a complete set of standards, this will continue to be a major headache.

If you’re looking for a powerful Web-site-creation program with the depth and breadth of a PageMaker or Director, NetObjects Fusion 3.0 should fill the bill nicely. Fusion goes about as far as a program can in hiding the complexities of HTML code. It also integrates well into environments that require some manual coding. Though it falls short of its goal of creating a universal code for all browsers, it’s one of the best solutions for nonprogrammers who want to use advanced techniques.

Version 2.0 of Netmind Services Inc.’s Enterprise Minder, an e-mail-based system that monitors Web sites for changes, includes notably improved monitoring options and management features.

PC Week Labs found the new features in Version 2.0, released earlier this month, to be welcome additions, providing users with several ways to track specific changes in Web sites, including a number-monitoring feature that proved useful for tasks such as tracking changes in prices at online stores.

However, some of the problems we found in the previous release, Version 1.04, are still there, including a propensity to fill users’ mailboxes with messages. But Enterprise Minder is still a unique, worthwhile application for businesses that need to gather up-to-date information from a variety of Web-based resources.

Enterprise Minder 2.0 is priced ranging from $45 per user for a 100-user license to $10 per user for a 10,000-user license. The server portion of the product runs under Windows NT and Solaris, and all administration and client access are done with a standard Web browser.

The Enterprise Minder application is essentially a central server that users access through a browser, selecting the sites they want to keep tabs on and the kind of changes they are looking for. Options include being notified of any changes in a Web page, changes in specific text areas of a page, changes in images and links, or the appearance of specific keywords. Version 2.0 also can watch for specific changes in numbers on a page: For example, notification can be sent when a product’s price drops below a certain amount.

Enterprise Minder informs users of the changes through an e-mail message. Notification can include an attachment of the Web page and, for intranet pages, can show the actual text changes in the page. Notifications can be sent upon detection of a change, once a day, once every two days or once a week.

We could enter groups of Web pages to monitor, or we could enter each page individually.

Monitoring changes

If a user chooses to monitor for any changes, the process is unchanged from the previous version. However, the options for advanced change monitoring are much improved in this version. For example, the advanced features previously required some understanding of HTML. Users can now choose these options simply by cutting and pasting from a Web page.

We especially liked the way the number-monitoring feature scanned all numbers from a page and made it simple to select which numbers to monitor. Other new features allowed us to monitor password-protected sites and dynamically generated Web pages. Version 2.0 does this by recording entries in a Web-based form.

However, Enterprise Minder still tends to quickly fill mailboxes with notification messages. This was especially true when we wanted to be immediately notified of changes. We would like to have the option of receiving a single e-mail message that lists all sites that have changed.

New administration features include a handy browser-based database query tool. Enterprise Minder 2.0 also includes new user template options that allow businesses to define basic settings for groups of users and enable easy implementation of changes across the organization.

Unfortunately, these templates can define only suggested settings. Users still have final control over their own options. Administrators should have the option to lock all Enterprise Minder settings for groups of users.

Enterprise Minder makes it easier to monitor Web page changes.

Summary

Although it boasts no major changes, NetMind’s updated Enterprise Minder has enough new features and improved capabilities to make it a worthwhile upgrade. Businesses will find it a good way to watch for changes in Web sites, but users must learn to define notification settings to limit the potential wave of e-mail messages the product can generate.

Pros: Improved options for monitoring Web pages for specific changes; can monitor protected Web sites and dynamically generated Web pages.

Cons: Not enough options for controlling user settings; needs more notification options to reduce load on user mailboxes.