Fidonet notes

From Driscollwiki

Jump to: navigation, search


Contents

Writing about Fido since Jan25 Egypt

Say hello to Fido

Beautiful Fido BBS advertisement

Fidonet on the Internet timeline: http://www.textfiles.com/fidonet-on-the-internet/tl.htm

Resources

Interaction w/ Internet, USENET

Store and forward communication: uucp and fidonet

Anderson, D. (2005?). "Store and forward communication: uucp and fidonet". Retrieved from: http://www.cs.cmu.edu/~dga/15-849/store_and_forward.html

Advantage of store and forward networks

  • "resilience to extreme disconnection"
  • messages can transmit in complete absense of end-to-end path
    • different from conventional e-mail implementation

Weakness: scalability

  • Fidonet nodes had to have updated list of ALL NODES
  • UUCP "bang" addressing was soon nightmarishly long

Protocols

"Both UUCP and FidoNet suffer from a somewhat murky layering model."

UUCP

  • various, difficult to understand protocols with diff features

Fidonet

  • conventionally XMODEM
  • no mention of Kermit in the Fidonet documentation even though it was developed in 1983
    • Evidence of parallel universes, XYZMODEM / Kermit

Naming scheme

Fidonet

  • Geographically hierarchical
  • Zone:Net/Node.Point(Subnode)
    • e.g. 1:512/66.0

UUCP

  • Globally unique hostnames
  • Character set limitations
    • e.g. "ucbvax", "mit-ai"
  • Assumes very small number of nodes

Routing

Fidonet

  • Routes determined by geographic proximity
    • Goal 1: reduce long-distance charges
    • Goal 2: maximum batching, max compression
  • Nodelist identifies every other system

UUCP

  • Source routing
  • Bang-path
    • e.g. "foo!bar!ucbvax!user"
    • Absolute pathnames
    • Very inconvenient for human end-user
  • All hosts needed an up-to-date map of the network

Scheduling

Fidonet

  • Requires nodes to be available during "Zone Mail Hour"
  • Additional transfers acceptable but might be limited by other user activity

UUCP

  • Schedule on "per-pair basis"
  • No global coordination

Higher level services

UUCP

  • Generic "file copy and command execution" protocol
    • Uses 'X.*' file to pass along a script for remote execution

"Modern UUCP"

  • Used to batch process queued mail on frequently offline servers

Addressing, naming, and routing

UUCP

  • Original design combined these into one "unfortunate" email address:
    • complex: hosta!hostb!hostc!user
    • sender-relative: hostx!hosty!hostz!user could be the same thing from diff machine

Pathalias

  • Mail program to compute the route based on a simpler address
    • e.g. hostc!user
    • or Fidonet e.g. 1:170/918:42

A Challenge

"For instance, an interesting exercise would be to devise a store-and-forward messaging system that runs on top of Internet protocols, does not assume constant connectivity, but still permits indirect, human-friendly naming that abstracts identity from topology."

FidoNet: Technology, Use, Tools, and History

Bush, R. (1992-3). "FidoNet: Technology, Use, Tools, and History". Retrieved from: http://www.fidonet.org/inet92_Randy_Bush.txt

Technical overview

Fidonet is first built for MS-DOS

  • "Semi-formal" data format, protocol specs
  • Ported later to UNIX, Apple //, etc.

Protocol design goal: Minimize long-distance

  • Contrast w/ uucp "g" protocol, smtp, nntp

Experimental IP implementation

Goal: traverse continents, avoid int'l dialing

  • EUNet, RIPE
  • EU, North American, Taiwan, Southern Africa, Chile, and others

Gateways

"1985.09.30: Bob Hartman announced that the UN*X gateway can send messages into Fidonet as well as to Fidonet from Usenet. The gate's address was 132/100." (http://www.textfiles.com/fidonet-on-the-internet/tl.htm)

Fidonet / uucp (thus Internet)

  • Fido addressing translated like this:
    • 1:105/42 == f6.n105.z1.fidonet.org
    • individual users addressed like this: randy.bush@p0.f42.n105.z1.fidonet.org
"Internetwork gateways have been used extensively by non-governmental organizations (NGOs) in Africa, as well as by an ingenious transport between the South African academic IP network (UNINET-ZA) and the Internet [Guillarmod 92]."
  • F. Jacot Guillarmod, "From FidoNet to Internet: the

evolution of a national network", "Proceedings of INET'92", H. Ishida Editor.

  • Murray, Janet, "K12 Network: Global Education through

Telecommunications", "Proceedings of INET'92", H. Ishida Editor.

Shared USENET and Fidonet echomail

  • Fidonet MODULA-2 echomail == USENET comp.lang.modula2
  • Fidonet K12Net == USENET k12.*

Internet / Fido connections

"FidoNet is a trademark of Tom Jennings and Fido Software. UNIX is a trademark of the soul-less [sic] behemouth AT&T.", Randy Bush, 1992


Common Software Components

BBS

  • Provides mail, news user agent ("M/NUA")
  • Users connect via dial-up from home PCs
  • Assumes not specific terminal software

Alternative: Standalone Editor

  • Console-based M/NUA
  • Useful when no BBS is available

Packer (or Scanner)

  • Mail/news transfer agent ("M/NTA")
  • Standardizes format of local data for transmission
  • Makes automated routing decisions
    • Usually based on local "routing rule file"
  • Uses echomail rule base to determine which groups to exchange with which other nodes

Mailer

  • Makes, receives Fidonet calls to/from other nodes
  • Handles modem init and operation
  • Implementation of transfer protocol (e.g. XMODEM)
  • Differentiate incoming human v. automated(Fido) calls
  • Schedule calls around timezones, pricing
  • Inter-node compatibility depends on mailer compatibility

Nodelist compiler

  • Transforms nodelist from a standardized FTS-0005 format into something local
  • Telephone numbers need to be adapted to local rules
    • Especially international dialing

Policy and politics

Fidonet is distinct from uucp, Internet

  • Low cost of entry
  • Nodes owned + op'd by individuals, hobbyists, users
    • Not computer professionals, corp labs, univ, mil/gov institutions
  • Without the implicit/explicit hierarchy/rules/norms of the mil/edu/com institutions, social/political issues were raised sooner on Fideonet (CITE THIS)
"Tom Jennings intended FidoNet to be a cooperative anarchy to provide minimal-cost public access to electronic mail. Two very basic features of FidoNet encourage this. Every node is self-sufficient, needing no support from other nodes to operate. But more significant is that the nodelist contains the modem telephone number of all nodes, allowing any node to communicate with any other node without the aid or consent of technical or political groups at any level. This is in strong contrast to the uucp network, BITNET, and the Internet."

Fidonet policy and conventions

1985, first policy document

  • http://www.rxn.com/~net282/POLICY1.txt
  • Technical
    • All nodes must send and receive
    • Established "National mail hour"
    • Established hubs, and nodelist coordinators
    • Restricted routing traffic thru unsuspecting nodes.
  • Social
    • No warez, illegal stuff
    • "Do not be excessively annoying and do not become excessively annoyed."

IFNA debacle

  • One reason this might have failed is that Fidonet was "more socially oriented" than technically (as was the case with uucp, etc.)

Policy-4

Written by nodelist coordinators

  • Establishes hierarchy of coordinators

Node-to-node architecture made radical takeover or top-down control impossible

  • Though fractured Fidonet would be a sad thing and sacrifice much value
  • Bad leaders generally ousted through (painful) social means
"The net runs fine, there is no need to implement bad policy just to satisfy some bureaucratic urge!", Tom Jennings, http://www.textfiles.com/bbs/FIDONET/JENNINGS/STANDARDS/policy4.art.txt

Transfer protocols (XMODEM)

"(de facto) standard file transfer protocol", Jennings 1983, http://www.textfiles.com/bbs/FIDONET/JENNINGS/STANDARDS/filexfer.doc.txt

History

Developed by Ward Christensen

  • 1977
  • Later developed first BBS
  • "X": transfer
"People who suggest I make SIGNIFICANT changes to the protocol, such as "full duplex", "multiple outstanding blocks", "multiple destinations", etc etc don't understand that the incredible simplicity of the protocol is one of the reasons it survived to this day in as many machines and programs as it may be found in!", Ward Christensen (Forsberg, 1985)

Purpose

File transfer over telephone lines

  • Electric noise yields data transmission errors
  • Protocol must detect errors, resend missing data
  • Asynchronous parity error detection
    • Not implemented by most IBM PC comm programs

Protocols do not work until they are standardized and implemented across organizations

"It was a quick hack I threw together, very unplanned (like everything I do), to satisfy a personal need to communicate with "some other" people. ONLY the fact that it was done in 8/77, and that I put it in the public domain immediately, made it become the standard that it is.", Ward Christensen, 1985 (Forsberg, 1985)

XMODEM protocol

(See: Figure 1, http://www.textfiles.com/programming/xmodem.txt)

  • When ready to receive, Receiving computer begins to transmit NAK (Negative Acknowledgement) character every 10 sec until Sender initiates transfer
    • After 9 tries, protocol must begin again
  • Sender sends the first packet:
    • SOH, Start of Header
    • Two block numbers
    • 128-byte block of data
    • Error-checking checksum
      • Remainder of (sum of ASCII values of each character in the block)/(255)
  • Receiver calculates its own checksum, it matches, it sends ACK, Acknowledge
    • If it fails, it sends NAK, for a resend
    • 9 sequential failures and the file transfer is aborted
  • When all data is sent, Sender transmits an EOT, End of transmission, character

XMODEM advantages

  • XMODEM is in the public domain
  • Easy to implement in BASIC and Pascal
    • Notably NOT C, lingua franca of UNIX
  • Only requires 256-byte comm receive buffer
    • Valuable for home pc users on 64k systems
  • Enables transmission of non-ASCII 8-bit data files (COM, EXE) common on IBM PC platform
  • Better error checking, 99.6% effective
    • Especially important for noisy telephone lines
  • Widely used on CP/M BBS
    • http://en.wikipedia.org/wiki/CP/M
    • OS for 8080/8085 and Z80 based micros
      • IMSAI 8080, Osborne 1, Kaypro, MSX, Altair, C/128, Amstrad CPC, ZX Spectrum, (BBC Micro, Apple II w/ z80 boards)
    • Standard platform in the MITS Altair mode

XMODEM/CRC, YMODEM

Forsberg, C. (ed.) (1985). "XMODEM/YMODEM PROTOCOL REFERENCE". Retrieved from: http://www.textfiles.com/programming/ymodem.txt

  • Available via uucp using "bang" path
  • Author contact info includes Voice, Modem, Usenet, Compuserve
  • Forserg educated at Western Military Academy, 1962
  • Also Ham (WA7KGX) and licensed aircraft pilot
"[XMODEM] is supported by every serious communications program because of its universality, simplicity, and reasonable performance."

Weaknesses in original XMODEM

  • Short blocklength
    • Slow thruput on timesharing, packetswitched networks (UNIX) and satellite circuits, buffered/error-correcting modems
  • 8bit checksum could be improved
  • Only one file per transfer
  • Transmitted file accumulates as many as 127 extra bytes
  • File meta data (modified) lost

XMODEM remained popular because

  • Widely implemented
  • Public domain
  • Well documented

YMODEM

Adhering to XMODEM principles

  • Simple
  • Public domain
  • Portable across systems
    • Many PC implementations
    • Many timesharing UNIX implementations

Improvements to XMODEM:

  • 1024 byte packet length
  • Batch protocol
  • Max throughput with error-correcting media (newer modems)
  • CRC-16 (16 bit checksum), CRC-CCITT
    • Same error correction as Bluetooth (see wp page)
    • x16 + x12 + x5 + 1
    • Refers to Tanenbaum's text Computer Networks (Forsberg, 1985)
    • Note: Tanenbaum also wrote Minix, the inspiration for Linux
  • Backward compatible
    • Sends preliminary <C> first
    • Four tries, 3 sec apart
    • If it fails, it defaults to traditional XMODEM style

Written in C

  • Compatible programs also written for UNIX
    • sb, rb: rbsb.sh

Kermit v XMODEM

http://www.columbia.edu/kermit/kermit.html

"Performance compromises and moderate complexity have limited the popularity of the Kermit protocol, which was developed to allow file transfers in environments hostile to XMODEM." (Forsberg, 1985)

Pre-Fidonet sketches

Jennings, T. (1984). Retrieved from: http://www.textfiles.com/bbs/FIDONET/JENNINGS/STANDARDS/fidonet.doc.txt

"Points that need ideas":

  • How to pay for it
  • How it will appear to callers trying to send mail
  • Mysterious operational type problems
"Like most of Fido, I imagine this too will be built with 90% suggestions."

Cost in 1984

"Right now, late night long distance (coast to coast) toll charges are about $13.00 per hour. ATT is proposing, as part of some other boring issue, lowering this to $10.00 per hour. This is quite cheap; a lot of messages can be sent in an hour."

Need an error checking protocol

  • But can't use XMODEM because transfer times "routinely double with long phone lines (Don't want to pay for that)"

Working out cost per message:

  • Sample message from LA to NYC = 0.01324$
  • Assuming
    • 80 character message
    • 1200 baud
    • 10% overhead for error correction

Why is cost so important?

  • BBSes are run independently
  • They don't make money
  • They usually cost money
  • If Fidonet is difficult or expensive, it won't be implemented
"IF YOU HAVE ANY IDEAS AT ALL ON THIS SUBJECT PLEASE LET ME KNOW!!!!!! THIS WILL MAKE OR BREAK THE FIDONET IDEA!!!!"

Network

"The topography of FidoNet is in keeping with bulletin board philosophy; totally random and as little organization as possible."

Initial design: Fidonet Electronic Mail Protocol

Jennings, T. (1984). "Fidonet Electronic Mail Protocol 20 Oct 84". Retrieved from: http://www.textfiles.com/bbs/FIDONET/JENNINGS/STANDARDS/fidomail.doc.txt

Background

Fidonet designed for portability

  • "within the existing 'public domain' software base"
  • any system that supports XMODEM should support Fidonet

Definition of Fidonet, 1984

"FidoNet is a true electronic mail system supported by the Fido Bulletin Board System software, under MSDOS version 2. It's function is to transfer textual messages, and binary files, between physically seperate computers on an automatic, unattended basis."

Platform assumptions

8 bit bytes

  • common to MS-DOS, CP/M

Transaction-oriented comm

  • Modem + phone line

Goal: "extremely short channel connect times"

  • Lowest possible cost
  • Thus, all packets pre-assembled
    • "No processing at all is done at packet transfer time."
      • This is dif from XMODEM checksum or YMODEM CRC

Non-real-time operating system

  • No daemon option like nntpd
  • Users connect to (presumably) Fido BBS (or other) to write + store messages
  • FidoNet later handles them, separate program, separate process
  • Once-per-day

Software assumptions

Fido/Fidonet is written in C w/ structs, strings

  • Characters: 8bit byte, 128ASCII
  • String: 1-dim array of 8bit chars, terminated by null (hex 0)
  • Integers: 16 bit, "short" ints
  • Longs: 32 bits, intel format

Packet definition

Treated as byte streams

  • Diffult to read ahead and buffer

Header

  • Fixed length
  • Originating, destination nodes
  • Date
  • Version number
  • Extra space for future expansion

Message 1...N

  • Fixed num of fields
  • Variable length
  • type: short
  • attribute: short
  • originating node: short
  • destination node: short
  • 00: short
  • cost: short
  • date: string
  • To: string
  • From: string
  • text: string

Trailer

  • Fixed length
  • Single integer 0
  • XMODEM might add data after this zero but it can be ignored

System assumptions

Borrows heavily from Ward Christensen's MODEM

  • XMODEM: mail packet transfer protocol
  • MODEM7/TELINK: transfer of any addition files and MSDOS file attribute info

Clock

  • Assumed and used for "failsafe monitoring of packets" and cost/call accounting
  • Important to avoid excessive (and COSTLY) broken connections

Three hardware dependent functions

  • DELAY(short), to sync with telephone company gear
  • MODOUT(char), send char via modem
  • MODIN(short), recv char via modem (wait n centiseconds)

Typical Fidonet session

Take control from Fido

Building routing table

Build list of active nodes

Creating packets

    • Compiling Messages
    • Compiling list of attached files

Sending/receiving Packets

  • Outgoing calls made at random 1-2 min intervals
  • Nodes selected from the previously build table
  • Upon connection
    • CRs sent to determine baud
    • Receiver sends signon
    • Sender transmits a TSYNC indicates call is from a FidoNet node
    • Receiver goes into XMODEM file receive
    • Sender transmits using normal XMODEM protocol
    • Receiver enters MODEM7, awaiting extra files
    • Sender transmits all files using MODEM7
    • Sender transmits EOT
    • Sender delays 5s
    • Sender disconnects
  • Upon successful transmission
    • Sender deletes outgoing packet, file list, and marks node successful
    • Returns to wait mode

Deleteing outgoing packets, marking sent messages

Unpacking incoming packets

Return control to Fido

Fidonet dialing and packet transfer

Note: at this stage, still requires enormous number of long-distance calls!

Note: numerous checks for carrier loss in code

  • numerous reasons for carrier loss

Jennings includes both pseudocode and C code w/ comments

Response: Proposal to mimic uucp in Fido

Wilkes, R. P. (1984). "FIDONET: Response 5/24/84". Retrieved from: http://www.textfiles.com/bbs/FIDONET/JENNINGS/STANDARDS/fidonet.rpw.txt

"People in the micro BBS environ often are totally unaware that there is a working, FREE, network of mini and microcomputers exchanging gigabytes of mail around the country (by phone). Some are part of the Arpanet, but the one we should examine is UUCP, a network of machines running Unix."

Proposed benefit:

  • Multi-hop routing a la uucp "bang"-style paths
  • Minimal, low costs
  • Connectivity with UNIX minis and other systems
  • Personal mailboxes
  • net.news (aka USENET!)
"Right now, you can mail a message and have it delivered free to almost any university or major technology corporation in the country via this network."

Wilkes tried to implement uucp on his BBS but:

  • Need for massive diskspace (33mb)
  • Need for multiple lines (he says at least 3)
  • Need faster processer than a 4.77Mhz 8086
  • Wilkes' conclusion: " ... wait."
"For the time being, I strongly urge that those that are strongly interested in this type of system start doing some research. When you can hold a reasonable discussion on file transfer protocols (real ones, of course--NOT XMODEM), message headers and formats, routing algorithms, connectivity analysis, delivery systems and scheduling, plus some of the more intricate cost analyses, we can join the work that is already advancing in the "other world" so we are not left out once again."

Response in 1989

Tobias, D. (1989). FidoNews. (TODO fix this cite) http://www.textfiles.com/fidonet-on-the-internet/878889/fido0624.txt

"He had fairly elitist expectations due to his involvement in UUCP/UseNet, and didn't see much value in a BBS network of much lower functionality. Fortunately, others were willing to work with what they had, and accomplished a lot with a network at not quite as lofty a level as Wilkes would have liked.", Daniel Tobias

Echomail

"In February 1986, Jeff Rush developed FidoNet's form of enews called echomail. As very few FidoNetters were familiar with the Usenet, they were quite surprised at the popularity and rate of growth of echomail. Within two weeks, an international echomail conference, MODULA-2, was propagated between Europe, Australia, and North America, and today the daily volume of compressed echomail is over eight megabytes. The social effects, both good and bad, of echomail on the network parallel those of the Usenet." (Bush, 1992-3)

Wager, W. (198?). "History of echomail." Retrieved from: http://www.textfiles.com/fidonet-on-the-internet/history/echomail.htm

Hartman, B. (1986). "FTS-0004: EchoMail Specification: The conference mail system." Retrieved from: http://www.ftsc.org/docs/fts-0004.001

First echoes

  • SYSOP
  • TECH
  • GAYNET
  • CLANG (C programming language)
  • BIBLE

(Wager, 198?)

Differences from USENET

  • Sysops have to explicitly subscribe to new groups
  • Binary attachments are ok

(Wager, 198?)

How it Works

  • Message is entered by a user dialed-in to a BBS
  • Message is exported with control information and linked to a conference
  • Receiving systems import the new message when they connect to the originating system
  • Receiving systems then export this message along with their own new messages upon future connects
    • Steps are taken to avoid redundancy and looping

Message control info

  • AREA:CONFERENCE, to which conference does it belong?
  • Tear line ('---'), may include product-specific information
  • Origin, Name of the BBS (Fido routing info)
  • Seen-by, list of Fidos that have seen this already, REQUIRED to avoid redundancy
  • PATH, list of machines that have actually processed this msg, rather than those to whom it has been sent

How echomail is sent

A. Along as FidoNet messages

Obvious scaling issue

  • Conferences can include 100s of messages
  • Very slow, long connections
  • Doesn't adhere to the POINT of Fido, avoiding LD charges

B. ARCmail: attach an ARC archive to FidoNet mailer

  • Thom Henderson wrote ARC archiver
  • Suggested using a compressed file to limit transfer

C. Conference Mail System

  • Eliminates ARCmail redundancies
  • Builds ARC during "export" phase
  • UnARCs them during "import" phase

Conference topology

Goals:

  • Increase Efficiency
  • Decrease Redundancy

Efficiency guidelines

  • Least time for all systems to recv msg
  • Least cost for all systems to recv msg
  • Fewest phone calls reqd for all systems to recv msg

Best efficiency practices

  • Never have two systems send to each other simultaneously
  • "Star" configuration, (node/hub)
  • Utilize fully connected polygons with a few vertices
  • Regional Echomail Coordinators (RECs) only allowed to make inter-regional connections

Jennings politics, views on hobbyists

"The FidoNet is militantly anti-commercial."
"FidoNet sysops are paranoid, cranky and take offense at the slightest mistake. You're better off banging hornets nests with sticks if ya wanna wrastle. Some have egos the size of small planets. Don't say I didn't warn you. Our computers may be smaller, but they are mean and smart. o FidoNet is organized in a radically different manner than nearly any other network you can think of. There is *no* central authority of any kind. You must deal with actual individual humans. Organization varies radically with time and place. Act as you would like to be treated. "

Other issues not addressed in the present investigation

Fido routing scheme

RECV-Only

  • No outgoing calls
  • No long-distance charges!
  • Unreliable but available option

Fido / Ham connections

"While the original FidoNet systems were fully integrated within bulletin board systems, FidoNet "mail-only" systems are now a noticeable portion of the public network. These provide the owner a facility similar to ham radio or a fax machine, but provide no public access via dial-up."


Retrieved from "http://www.kevindriscoll.org/wiki/Fidonet_notes"
Personal tools