Document: fsc-00xx
Version: 0.6
Date Aug 30, 1995
Title: Implementation and Usage of FileFind Utilities
Authors: Robert Williamson FidoNet#1:167/104.0 robert@ecs.mtlnet.org
Intro
A portion of the document is derived from information in
AllFix.DOC by Harald Harms @ 2:281/910
with additional sections from
FQuery.DOC by Robert Williamson @ 1:167/104
The MSdos program ALLFIX by Harald Harms first introduced the idea
of searching for files via echomail. The term applied to this function
is 'FileFind'. A FileFind system allows sysops, points and BBS users
to search for files by placing a message to 'ALLFIX' in an echo
designated for the purpose of finding files. All FTN sites running a
FileFind processor which is configured to scan that echo will reply to
that user if there any files matching his query. This system provides
a method for searching many FTN sites throughout the world, with a
single message.
FileFind programs work by either scanning through defined message
bases or scanning packets for defined AREA tagnames for messages to the
default name ALLFIX. All FileFind programs MUST respond to the name
ALLFIX, but may also respond to the name FILEFIND and the name of the
particular FileFind program in use or defined for the echo. The
FileFind program will process these messages, examining the Subject
field for search queries. If any valid query is found, the FileFind
program will search the sites files database for files matching the
users's query.
If the FileFind program finds any matches, it will generate a reply
containing a list of the files found, and some basic information ABOUT
the system posting the reply. When the user who initially wrote the
request reads the reply, he will then be able to decide if any of the
reported files meet his needs, and from the ABOUT included in the
reply, learn where and how he may get those files.
FileFind Query Message Structure
To: name_of_FileFind program
The message must be addressed to ALLFIX so that all FileFind programs
can respond. To use features specific to a particular FileFind
program, or to limit the responses to a particular platform, the
message should be addressed to that program's name. Some FileFind
programs will respond to more than two names.
Subject:
A space-separated list of file specifications, keywords or quoted
strings.
keyword - single word preceeded by a '/' with no intervening spaces,
must be at least 3 characters, not including the '/'.
a keyword search is in actually a substring search of the
site's filelist.
description - string enclosed in double-quotes,
if a single word, must be more than 3 characters.
filespec - single word, no spaces, no double-quotes or preceding /,
must be at least 3 characters, not including any wildcard
or pattern matching charcaters, such as '*'.
Messages addressed to ALLFIX must not have any embedded
pattern matching characters.
The minimum number of characters for description, keyword and
filespec queries is an implementation detail of the FileFind program.
These values should be configurable, but should never be settable to
values of less than 3.
Each implementation should allow the operator the ability to
configure a list of disallowed keywords.
NetMail Queries
Some FileFind programs may also have the ability to process file
search queries received as netmail and addressed to the name of the
particular FileFInd program with this capability. In this case, all
replies are via netmail also.
NetMail Commands
FileFind Netmail commands are identifed by a leading '%'.
Implementation of netmail commands is optional. If implemented,
compliant FileFind utilities should be able to process the following
minimum NetMail command set.
%HELP - netmail only, returns an extended help text for the
FileFind program, the ABOUT of the the site and a list
of MAGIC freqable names.
%ABOUT - netmail only, returns the ABOUT of the site and a full
or %MAGIC list of MAGIC names.
%NEWFILES - netmail only, returns the NEWFILES list of the site
or %NEW via netmail.
Extended NetMail Commands:
Implementation of the following netmail commands is optional and
not required for compliance with the FileFind NetMail Command set.
%REPORT
- sends a configuration report for echo
this allows an echo moderator to check if a site running
a FileFind utility is compliant with the rules of the
filefind echo.
%REQUEST
- if found, will place requested file on hold for remote
site
%UUREQUEST
- if found, and the filesize after uuencoding is less
than 60K, it will be sent as multiple netmail messages
The Site ABOUT
Obviously, a system that neither accepts file requests nor allows
users to download on their first call should not be responding to
FileFind messages. If there are any limitations for the caller to
acquire any of the files that the site has advertised as being
available in it's FileFind response, these limitations MUST be listed
in the reply. This information should be included in the ABOUT file
that the FileFind program user creates.
The site ABOUT should contain the following information. The
FileFind program implementor should instruct his users on these
requirements.
- sitename
- site operator's name
- complete phonenumber
- baud rate
- hours during which filerequests are accepted, if at all
- hours during which users can download
- conditions for file requests and user downloads
NOTE: the above information should be within the first 14 lines.
optional:
- a list a MAGIC names
- an indication if magic names are also available to terminal users.
Searching for Files and Creating Replies
The method used by the FileFind program to search for requests is
up to the implementor. However, if searching a list, the FileFind
program should confirm the actual existance of all files that match the
query specification.
The FileFind program should only process description strings,
filespecs or keywords that contain more than 3 valid characters and
should have configuration options to define greater minimum lengths on
a per-echo basis.
For filespecs, the wildcard character '*' IS considered a valid
specification as well as the '?' wildcard, but only the '?' is to be
counted as a character when determining the length of query. File
extensions are not necessary and any characters AFTER a '*' are to be
ignored. The FileFind program should be configurable so as to allow
replacement all of the file extensions with '.*' or '#?' dependant on
platform. This results in queries being independant of the various
archivers in use.
Replies
Replies created by FileFind utilities are expected to be in
compliance with the following FTN specifications:
FTS-0001 - packed message format
FTS-0009 - MSGID/REPLY
FSC-0046 - PID and tear line
In addition, a FileFind utility may use the FID: control line for
any information needed that cannot be put in a PID: without violating
that specification.
^AFID: ascii text CR
Must be less than 80 characters including ^A and terminating CR.
There are three ways in which the FileFind program can create replies:
- write the replies in the echo in which the query appeared.
- write the replies in an echo that has been specifically
designated for that purpose in the particular FTN or for
a gorup of echos in that FTN.
- reply via routed netmail.
Since each FTN site connected to a particular FileFind program area
is capable of creating an information reply, there is much concern as
to the amount of traffic that can be generated, FileFind program
developers must be sensitive to these concerns by providing the means
to their users to limit the traffic on a per-echo basis. For example,
various FileFind echos have rules limiting the size or number of
replies, or the length of the system information that may be included
in a reply.
Limiting replies
It is strongly suggested that some default limitations be built-in.
Limiting Site Header (ABOUT):
If the site's ABOUT, (the text that has been configured in order to
add the system's information and Magic names list to the reply), is
greater than 14 lines, the remainder should NOT be posted. A line
should be added to the response indicated this, and the user may be
invited to either Freq or download the MAGIC name's ABOUT or MAGIC, for
a full list of magic names. The FileFind program may optionally send
the full system information and magic name list via routed netmail.
Limiting Match List due to ambiguity of query:
If the list of matches (note: not the size of the message itself)
is greater than 32K, the FileFind program should post a message to the
user to indicate that his query may have been too ambiguous and perhaps
invite him to freq or download the MAGIC name FILES for a full list.
Splitting Match List into Multiple Messages:
If the list of matches is greater than 10K, it should be split into
multiple messages of no more than 8K. Although the backbone permits
messages up to 16K in length, 8K is a more readable size. Only the
first split message may contain the ABOUT information of the site.
Each message must be given both a unique Subject field (eg: prepended
by "Part n/n") and a unique MSGID:. This because some tossers may use
either or both for dupe detection.
Limiting Number of Split Messages:
If the number of messages is greater than the preset limit of the
echo, and the FileFind Program does not have an option to forward the
replies via netmail, the replies should be discarded and the user
informed that his request may have been too amibiguous.
NetMail Reply:
The FileFind program may have an option to forward all replies via
routed netmail, or to do so under certain conditions as outlined above.
Obviously, if the FileFind program can process netmail queries, it MUST
respond via netmail.
User NetMail Reply Request:
Alternativly the user can request a netmail reply for his echomail
query by preceeding the query with either "%" or "!".
eg;
Subject: % /fsc /fts
If the FileFind program does not support this feature, it must
ignore any echomail query message that has a "%" or "!" as the first
WORD of the Subject field.
Second Reply or Extended Response Request:
The FileFind site indicates availablility of Second Reply by
placing the string 'program_name 4d_address' in the From: field of the
message.
eg: FROM: FQUERY 1:167/104.0
When a user replies to a FileFind reply, the message will be to the
FileFind program @ {network address}. When processing the FileFind
conferences, the FileFind program will treat any message to itself that
includes the site address as a Second Reply Request.
If this feature is available, the FileFind program will include up
to a maximum of 15 files (maximum 12K match list) in it's replies. If
the user wants a more detailed listing, he simply replies to the
FileFind program's reply. Only the system that posted the original
reply will respond to that new request. This second, specific reply,
will contain up to 50 files (32K of matchlist) either including or
SKIPPING the first 15. These numbers may be replaced by byte limits in
some implementations.
No Second Reply in Designated Reply Echo:
The Designated Reply Echo method does not allow replies to be made,
because the FileFind program may not be permitted to scan a Designated
Reply Echo. The FileFind program should automatically report up to 50
files for any requests. Therefore, the traffic limitaion features may
be disabled for networks that require the FileFind program to reply in
a Designated Reply Echo, and disallow Second Reply in that echo.
Disable Local Messages:
The FileFind program must be able to to disable the processing of
local messages. What this means is that the FileFind program will not
process any messages generated on that FTN site, including messages by
the sysop using an offline reader, or by a site's BBS or off-line
reader users. This should NOT exclude messages from a site's points.
Limit by Age:
The FileFind program must be configurable so that the operator can
limit the age of an query message that is acceptable for processing.
This should be in number of days. The FileFind program may be
configured to process all the FileFind requests regardless of how old
they are. Age should never be greater than 365 days.
LinkMGR Support:
Implmentors may choose to support the LinkMGR proposal for netmail
queries and commands. In this proposal, the queries and commands do
not appear in the subject field but rather, in the the BODY of the
message. The subject field wil contain the LinkMGR password.
Use of the LinkMGR method allows the user to send multiple commands
to the fIleFind program.
Go Back