This is an old revision of the document!


QWK Format

QWK packets and their closely related Reply (REP) packets are single-file archives (usually in PKZIP format) containing one or more data files. QWK packets are filenames with a .QWK suffix/extension, while Reply packet filenames have a .REP suffix.

Packet Naming

A QWK packet filename is normally of the form: ID.QWK (or ID.qwk), where ID is the unique identifier (up to 8 valid MS-DOS filename characters, starting with an alphabetic character) of the system from where the messages were downloaded (received). This ID is commonly referred to as the “BBS ID” or “BOARD ID” in original QWK specifications and in Synchronet, often referred to as the system “QWK-ID” or “QWKnet ID”.

Similarly, a QWK Reply (REP) Packet filename is normally ID.REP (or ID.rep) where ID is the QWK-ID of the destination system for which the REP packet is to be sent (uploaded).

When multiple packet filenames with the same ID are stored in the same directory, it is customary to replace the last character in the filename suffix with an incrementing decimal digit, beginning with 0. e.g.

VERT.qwk
VERT.qw0
VERT.qw1
VERT.qw2

If there are more than 11 QWK packet filenames, then it is customary to replace the last 2 characters with incrementing decimal digits, beginning with 10, e.g.

VERT.REP
VERT.RE0
...
VERT.RE9
VERT.R10
VERT.R11
VERT.R12

QWK File

QWK packets are typically created-on and downloaded-from a large message archive (e.g. a collection of message bases) on a remotely-accessible system (e.g. BBS). The messages contained in the packet may include both private messages (e.g. e-mail) and public messages (e.g. posts on topical conferences). While the private messages are typically addressed *only* to the user that downloaded the QWK packet, the public messages may be addressed to anyone (or often, “All”).

The files that may be contained within a QWK packet (archive) include:

Filename Standard Description
MESSAGES.DAT QWK Binary file containing body text and limited header information for all messages contained in the packet (at minimum, a QWK packet must contain this file)
HEADERS.DAT Synchronet Text file (in .ini file format) containing detailed header information for every message in the packet
VOTING.DAT Synchronet Text file (in .ini file format) containing voting information (polls, ballots, up/down-votes) contained in the packet
CONTROL.DAT QWK CRLF text file containing metadata about the system (BBS) where the messages came from including the names of the conferences where public messages may have originated
DOOR.ID QWK CRLF text file containing metadata about the software that created the QWK packet
NETFLAGS.DAT QWK Binary file containing a 'net status' indicator (0x00 or 0x01) for each message conference
NEWFILES.DAT QWK CRLF text file containing list of newly added/uploaded files on the system that created the QWK packet
PERSONAL.NDX QWK Binary file containing pointers to messages addressed to the user that created / downloaded the QWK packet
000.NDX QWK Binary file containing pointers to the mail “inbox” for the user that created / downloaded the QWK packet
xyyy.NDX QWK Binary file containing pointers to messages in public message group x, sub-board yyy (Synchronet QWK conference numbering convention)
TOREADER.EXT QWKE CRLF text file containing metadata about the user that created / downloaded the QWK packet

The first 3 files (MESSAGES.DAT, HEADERS.DAT, and VOTING.DAT) are the most important when it comes to Synchronet QWK networking.

MESSAGES.DAT

The MESSAGES.DAT file (and ID.MSG file) is written in multiples of 128-byte blocks.

The first 128-byte block in the file contains a space-padded US-ASCII string identifying the software that generated the packet.

fprintf(qwk,"%-128.128s","Produced by " VERSION_NOTICE "  " COPYRIGHT_NOTICE);

Conversely, the first 128-byte block of an ID.MSG file from a .REP packet contains the QWK-ID of the system for which the packet was created and is intended to be uploaded-to.

  • All subsequent 128-byte blocks in the file are either header blocks or body blocks.
  • Each header block identifies a unique message in the packet.
  • Header blocks are fixed length (128 bytes).
  • Each message has only one corresponding header block.
  • A normal message will have at least one body block (at minimum).
  • Vote messages (ballots, polls, up/down-votes) will not have a corresponding body block (see VOTING.DAT file).
  • A header block includes an indication of the total number of blocks before the next header block (minimum: 1).

Header Block

ASCII string fields are space-filled and generally left-justified. Unused bytes are generally filled with a space (ASCII 32) character. Multi-byte integer are stored in little-endian byte order (LSB first).

Offset Length Format Description
0 1 ASCII char Message status (“*”, “+”, “~”, “`”, “%”, “^”, “#”, “$”, “-”, “ ”, or “V”)
1 7 ASCII string message number for QWK packets, conference number for REP packets (7 decimal digits)
8 13 ASCII string Date and time (“MM-DD-YYHH:MM”)
21 25 ASCII string To name
46 25 ASCII string From name
71 25 ASCII string Subject
96 12 ASCII string Password (unused by Synchronet and un-populated by Synchronet)
108 8 ASCII string In-reply-to message number (8 decimal digits)
116 6 ASCII string Number of blocks for the message, including header block (1+)
122 1 byte Message status: 225 = message active, 226 = message deleted
123 2 uint16 Conference number
125 2 uint16 Relative message number in packet (1+) (unused and not populated by Synchronet)
127 1 ASCII char Net tag line (“*” if present, “ ” otherwise)
Message Status

The Message Status character has the following known definitions:

Status Description
Public, unread
- Public, read
* private, read but someone, but not by intended recipient
+ private, read by intended recipient (up for debate)
~ comment to sysop, unread
` comment to sysop, read
% sender password protected, unread (unused by Synchronet)
^ sender password protected, read (unused by Synchronet)
! group password protected, unread (unused by Synchronet)
# group password protected, read (unused by Synchronet)
$ group password protected to all (unused by Synchronet)
V a vote message (poll, ballot, or up/down-vote) - Synchronet
Issues

As you may have noticed, the QWK message header block contains some narrow limitations and bizarre inconsistencies. For example:

  • Most fields are US-ASCII, while some are binary integers
  • Names (to and from) are limited to 25 characters while standard FidoNet messages support up to to 35-character names
  • Subjects are limited to 25 characters while standard FidoNet messages support up to 71-character subjects
  • The “password” field doesn't seem to have ever been used by anything, but easily could have been used to extend subjects to 37 characters
  • Message dates are limited to 2-digit years
  • Message times are in hours and minutes only (no seconds)
  • No timezone information
  • A maximum conference number of 9,999,999 is allowed in the field at offset 1 of a .MSG file, while the conference number at offset 123 is limited to just 65,535.
  • The message number at offset 1 of a MESSAGES.DAT file is limited to 9,999,999 while the In-reply-to message number (at off 108) can hold a number up to 99,999,999.

All of this craziness is addressed with the Synchronet-defined HEADERS.DAT file.

Body Block

The text of a message is contained in one or more body blocks which follow the header block.

If the message does not fit in an exact-multiple of 128-byte blocks, the last body block of a message is padded with spaces or NULLs.

For unclear reasons, new-lines sequences are normally represented by the character 0xE3 (227). In UTF-8 encoded messages, Synchronet uses ASCII 10 (LF) to represent new-lines in QWK message bodies.

The first lines of a message's text may contain metadata for the message (so-called “kludge lines”). Any blank lines following a set of kludge lines should be ignored.

QWKE Kludge Lines

The Extended QWK (QWKE) standard defines 3 initial lines which, if present, replace the equivalent values from the header block. The QWKE standard says that the “kludge lines” may be terminated by either the QWK newline character (0xE3) or an ASCII carriage-return (13) character. The QWK newline character (or ASCII LF for UTF-8 encoded message) is recommended.

Kludge Description
To: To user name (when > 25 characters)
From: From user name (when > 25 characters, not used by Synchronet)
Subject: Message subject (when > 25 characters)
Synchronet Kludge Lines

There are some additional Synchronet-defined kludge lines that may exist at the beginning of a message body:

Kludge Description
@VIA: Originating QWKnet address (full route)
@MSGID: Unique Message-ID (RFC822-style)
@REPLY: In-reply-to Message-ID (RFC822-style)
@TZ: Originating time-zone in SMB hexadecimal format
@REPLYTO: Name and address to which replies should be sent

All of this craziness is addressed with the Synchronet-defined HEADERS.DAT file.

HEADERS.DAT

As can be seen in the definition of the QWK MESSAGES.DAT header block definition and the subsequently defined “kludge lines” to address its problems, the QWK format was not really designed to be very extensible or accommodating of newer or larger header fields. However, because the QWK packet is an archive of multiple files, adding more standard files is relatively trivial, hence the “new” Synchronet-defined HEADERS.DAT file1).

The HEADERS.DAT is a text file (CRLF or LF-terminated lines is fine), in .ini file format. Through the use of the HEADERS.DAT file, long header field values and additional header fields may be defined and used without resulting to “kludge lines”.

Format

Each section in the HEADERS.DAT file correlates with a message in the corresponding MESSAGES.DAT (or ID.MSG) file. The section name is the byte-offset (in hexadecimal) into the MESSAGES.DAT file of the corresponding message's header block.

Key/value pairs may be specified using key = value or key: value syntax, but the latter is preferred to indicate to the parser that trailing white-space is significant and should be retained. Leading white-space in header field values is never significant and always ignored/stripped.

Character Set

Header field values may each be up to 1024 US-ASCII characters in length. When “high ASCII” characters are included in any header field values, the IBM CP437 character set is assumed unless there is an indication that the header fields are in UTF-8 format, in which case all header fields, for a particular message, with “high ASCII” characters must be in UTF-8 format. As with FidoNet messages, the message body text and header fields must share the same encoding. If the “Utf8” HEADERS.DAT header field value for a message is set to “true”, then all the headers fields and the body text of that message are to be interpreted as though they were UTF-8 encoded.

Example

The key: value pairs within each section of the HEADERS.DAT file correlates with the message at the corresponding byte-offset in the section name. So, for example:

[c80]
Message-ID: <5D4AFDF1.40645.dove_dove-gen@somebbs.com>
WhenWritten:  20190807093601-0700  c1e0
WhenImported: 20190807093601-0700  c1e0
WhenExported: 20190807095552-0700  c1e0

In this example, the Message-ID header field is the full RFC822-style message-ID associated with the messages in the MESSAGES.DAT a byte-offset 0xC80. The WhenWritten/Imported/Exported header fields provide detailed date, time, and timezone information about when the message was written, imported into the original system's message base, and exported out of the original message base (e.g. to a message network).

When a header field is present in the HEADERS.DAT file, it takes precedence over that same header field in either the MESSAGES.DAT header block or any kludge lines that may have been included in the message text (from the message's body blocks).

Header Fields

The following is a complete list of supported header fields within sections of the HEADERS.DAT file:

Field Description
Utf8 Header fields and body text use the UTF-8 character set (when true) rather than CP437 (the default)
Message-ID RFC822-style unique message identifier
In-Reply-To RFC822-style message ID of message that this message is a reply to, if relevant
WhenWritten Date/time (in ISO-8601 format) and timezone (in hexadecimal SMB format)
WhenImported Date/time/zone message was imported into the originating system's message base
WhenExported Date/time/zone message was exported from the originating system's message base
ExportedFrom Details from where the message was exported: system ID, message-base, and message number
Sender Message author/sender
SenderNetAddr Sender's network address (e.g. QWK-ID), if relevant (e.g. routed)
SenderIpAddr IP address of original author's system
SenderHostName hostname of original author's system
SenderProtocol Name of protocol used by sender (e.g. “Telnet”, “SSH”, “RLogin”, “SMTP, “NNTP”, “HTTP”, etc.)
Organization Name of sender's organization (e.g. BBS)
Reply-To The network address to direct replies to this message
Subject Message subject
To Message recipient name
ToNetAddr Recipient's network address, if relevant
X-FTN-AREA FidoNet AREA tag
X-FTN-SEEN-BY FidoNet SEEN-BY line
X-FTN-PATH FidoNet PATH line
X-FTN-MSGID FidoNet unique message identifier
X-FTN-REPLY FidoNet in-reply-to message identifier
X-FTN-PID FidoNet-style program identifier of originating system software
X-FTN-Flags FidoNet FLAGS kludge value
X-FTN-TID FidoNet EchoMail Program (tosser) Identification
X-FTN-CHRS FidoNet character set (charset) identification
X-FTN-Kludge FidoNet control paragraph (not explicitly supported with another header field)
Editor Program (software) identifier of message editor used to create original message text
Columns Number of terminal columns (if known) used by the author while composing message text
Tags Space-separated messages tags
Path USENET Path information
Newsgroups USENET Newsgroups (list)
Conference Conference number where message is to be (or was) posted

Additional keys may be included representing not otherwise specified Internet (RFC822-style) header fields.

VOTING.DAT

The VOTING.DAT file is a more recent addition to the QWK packet format (introduced with Synchronet v3.17) and it contains information about message-based voting:

  • Polls
  • Poll-closures
  • Ballots (votes on polls)
  • Message up/down-votes

Very similar to the HEADERS.DAT file, the VOTING.DAT file is a text file (with CRLF or LF-terminated lines), in .ini file format. A vote-message header block is identified in a QWK header block with a Message Status value of ”V“ (ASCII 86). The vote section of interest is located in the VOTING.DAT file by first locating the section with the corresponding header block byte-offset in hexadecimal (e.g. [f00]), and then immediately following that section label, will be another section that identifies the type of vote message and this section will contain the relevant key/value pairs:

Section Description
[poll:<id>] A posted poll with unique message-ID, id
[vote:<id>] A posted vote with unique message-ID, id
[close:<id>] The closure of previously posted poll

Polls

A VOTING.DAT poll section ([poll:<id>]) may contain the following keys/values:

Subject The polling subject/question/summary
Sender The name of the pollster
Conference The conference number on the target system
MaxVotes Maximum number of votes per ballot for this poll (between 1 and 16)
Results Visibility of voting results (0=voters, 1=open, 2=closed, 3=secret)
Commentn Comment line (0+) to display before available answers (optional)
PollAnswern Available answer (0+) to this poll

A poll is closed by a subsequent [close:<id>] vote-message-section which may occur at any later point in time (e.g. even in later QWK packets, received weeks or months later) and once discovered should mark the original poll (identified with the In-Reply-To key value) as unavailable for new ballot submissions.

Ballots

Ballot message sections are identified by a [vote:<id>] section. There are 2 possible types of ballots:

  1. Poll Ballots: contain a “Votes” key
  2. Message Ballots: contain an “UpVote = true” or a “DownVote = true” key/value pair

Poll ballots are to be applied to the open-poll referenced by the “In-Reply-To” value. If the poll has been closed, then the ballot is ignored. The “Votes” key value is a bit-field (normally in hexadecimal representation with “0x” prefix), indicating which vote options the voter has selected (bit-0 = PollAnswer0, bit-1 = PollAnswer1, etc.).

Message ballots are simply up/down-votes to be applied to a message's popularity score. The message being voted on is referenced by the “In-Reply-To” message-ID value. If the referenced message doesn't exist, the vote is ignored. The “Sender” key/value identifies the voter. Messages are never closed to new ballots, but voters should not be allowed to submit more than one ballot per message.

NDX Files

The .NDX (index) files contained in QWK packets are of a notoriously bad format. These files contain byte offsets into the MESSAGES.DAT file, which is fine. But these offsets are stored as 32-bit real numbers in an obsolete “Microsoft Binary Format”. Additionally, each record contains a conference number, but that conference number is only 8-bits, limiting its usefulness to only 256 possible message areas.

Some message readesr do not require or can be configured to ignore these .NDX files. There is no unique information stored in the .NDX files, so it's entirely possible for any consumer of QWK packets to be programmed to create their own useful index information on-the-fly (e.g. the first time the packet is opened).

Synchronet does not parse or use .NDX files in any way, although it can generate them (user's choice).

NETFLAGS.DAT

The NETFLAGS.DAT is of questionable usefulness or value. Synchronet creates this file only for QWK packets destined for QWK network nodes (all bytes containing 0x01). Synchronet does not read or use this file from incoming packets.

Bulletin Files

Optional bulletin files (in CRLF text format) may also be found in QWK packets. For example:

Filename Description
HELLO Displayed to reader of message packet
BBSNEWS Displayed to reader of message packet
BLT-* Displayed to reader of message packet
GOODBYE Displayed to reader of message packet upon exiting packet

Additional Files

Other files found in a QWK packet were usually placed there because the user requested them to be included (e.g. from a download queue) or they were attached to messages included in the packet.

REP File

REP (reply) packets are typically created by a user using an offline mail reader or by a system participating as a down-stream node in a message network using QWK packet technology (e.g. DOVE-Net).

The files that may be contained within a Reply packet (archive) include:

Filename Standard Description
ID.MSG QWK Binary file containing reply messages (format nearly identical to that of MESSAGES.DAT). The base filename is the unique QWK-ID of the system where the original QWK packet was created / downloaded-from
HEADERS.DAT Synchronet Text file (in .ini file format) containing detailed header information for every message in the packet
VOTING.DAT Synchronet Text file (in .ini file format) containing voting information (polls, ballots, up/down-votes) contained in the packet
TODOOR.EXT QWKE CRLF text file containing control instructions from the offline mail reader to the BBS

Original QWK format

Conference Names

The message area (conference) names included in the CONTROL.DAT file are only used for reader information/display purposes. The conference *numbers* are references used to identify specific message areas between the reader (or QWKnet node) and the host/hub BBS.

Length

According to “QWK Mail Packet File Layout” (qwklay) by Patrick Y. Lee (version 1.6 - December 19, 1992), the conference names are limited to “13 characters or less”.

According to “The Mysterious QWK-File Format” by Jeffery Foy (April 25, 1991), the conference names are limited to “10 characters or less”.

According to “QWKE Specifications 1.02” by Peter Rocca, “conference area names are no longer limited to 13 characters, they are limited to 255 characters now [in QWKE]”.

Extended QWK (QWKE) Format

See Also

1)
added to Synchronet in 2011
ref/qwk.1618348665.txt · Last modified: 2021/04/13 14:17 by digital man
Back to top
CC Attribution 4.0 International
Driven by DokuWiki Recent changes RSS feed Valid CSS Valid XHTML 1.0