Animated Logo Rietta Software
Home News Products Web Services Product Support Search
RoboGen - Visual Exclusion File Editor

Background

The Robot Exclusion Standard was proposed by Martijn Koster back in 1994; while it is not enforced by any organization, it is widely supported by web robots used by search engines.  Robots.txt (the filename used on servers to denote an exclusion file) is the web equivalent to a DO NOT ENTER sign.  While it does not keep someone, or a robot that is programmed to ignore this file, from viewing the contents of the files or directories you list, it does provide a convenient way to tell the search engines which part of your web site not to index.  For example, you are running a web store and you want your home page and catalog pages listed in search engines, however, you don't want the order form listed.  To keep the order form from being listed, you would simply add its path to your robot exclusion file and upload it to your server.

What RoboGen Does

So, how does RoboGen fit into all this?  RoboGen is a visual editor for Robot Exclusion Files; it allows one to create agent rules by logging onto your FTP server and selecting files and directories clicking the arrow button, <<<.  It also maintains a comprehensive list of user-agents which can be selected from a drop list. Here are some screen shots and features:
 

Ease of Use

Create and Manage robot exclusion files (robots.txt) with the unprecedented ease using features such as:
  • Setting varying User-Agent rules by dragging the files and directories from your FTP site.
  • Selecting user agents from a list of over 180 known robots.
  • Load and edit existing robots.txt files.
  • Use RoboGen to check your existing exclusion files for common errors and correct them.
Agent Menu

Fast Access

Use the Agent-Menu to switch between bot databases and to manage other User-Agent information.

Robot Databases

Use the Bot Manager to view and edit robot information.  The three separate sections include :
  • Known Robots which lists 177 known user agents.
  • Popular Search Engines, which list the user agents of the more well known search engines.
  • User Defined for you to add user agents of which you know.
 Bot Manager - Viewing Known Bots

Bot Manager Dialog

Customizability

Use the Header Editor to create the header which will be placed at the top of every file edited by RoboGen.  You can even use include statements such as ---timestamp--- which will automatically be replaced with the appropriate information at save time.		 Header Editor

 
Download Center
RoboGen Full Installation robogen-setup.exe (1.14 mb)
  RoboGen Zip File Distribution
Only download this if you are sure that the mfc42.dll file in your windows\system folder is version 6.00.8447.0 or greater.  You can download this version separately in a 400 kb zip file here.		 robogen-nomfc.zip (189 kb)

VERY IMPORTANT!
RoboGen was developed using the newest edition of Visual C++ and needs the newest Microsoft Foundation Classes library (mfc42.dll).  If you do not have the latest version of the Microsoft Foundation Classes then RoboGen will crash when you try to use it.  The full installation comes with this file.

Other Resources

RoboGen Message Board

RoboTag - JavaScript Edition

Robot Exclusion Protocol
 


Copyright ©1999 Rietta.  All Rights Reserved
Last updated: 6/22/1999

visitor since 6/14/1999.
FastCounter by LinkExchange