Differences

This shows you the differences between two versions of the page.

--- ref:qwk [2019/08/09 02:13] – [MESSAGES.DAT] Include Message Status definitions digital man
+++ ref:qwk [2023/12/12 20:54] (current) – [Original QWK format] Convert FTP links to HTTP digital man
@@ Line 1: / Line 1: @@
 ====== QWK Format ======
-QWK packets and their closely related Reply (REP) packets are single-file archives (usually in PKZIP format) containing one or more data. QWK packets are filenames with a ''.QWK'' suffix/extension, while Reply packet filenames have a ''.REP'' suffix.
+QWK packets and their closely related Reply (REP) packets are single-file archives (usually in PKZIP format) containing one or more data files. QWK packets are filenames with a ''.QWK'' suffix/extension, while Reply packet filenames have a ''.REP'' suffix.
+===== Packet Naming =====
+A QWK packet filename is normally of the form: ''//ID//.QWK'' (or ''//ID//.qwk''), where //ID// is the unique identifier (up to 8 valid MS-DOS filename characters, starting with an alphabetic character) of the system from where the messages were downloaded (received). This //ID// is commonly referred to as the "BBS ID" or "BOARD ID" in original QWK specifications and in Synchronet, often referred to as the system "QWK-ID" or "QWKnet ID".
+Similarly, a QWK Reply (REP) Packet filename is normally ''//ID//.REP'' (or ''//ID//.rep'') where //ID// is the QWK-ID of the destination system for which the REP packet is to be sent (uploaded).
+When multiple packet filenames with the same //ID// are stored in the same directory, it is customary to replace the last character in the filename suffix with an incrementing decimal digit, beginning with ''0''. e.g.
+<code dir>
+VERT.qwk
+VERT.qw0
+VERT.qw1
+VERT.qw2
+</code>
+If there are more than 11 QWK packet filenames, then it is customary to replace the last 2 characters with incrementing decimal digits, beginning with ''10'', e.g.
+<code dir>
+VERT.REP
+VERT.RE0
+...
+VERT.RE9
+VERT.R10
+VERT.R11
+VERT.R12
+</code>
 ===== QWK File =====
@@ Line 13: / Line 40: @@
 | ''HEADERS.DAT''      | Synchronet | Text file (in [[config:ini_files|.ini file format]]) containing detailed header information for every message in the packet |
 | ''VOTING.DAT''       | Synchronet | Text file (in [[config:ini_files|.ini file format]]) containing voting information (polls, ballots, up/down-votes) contained in the packet |
-| ''CONTROL.DAT''      | QWK      | CRLF text file containing metadata about the system (BBS) where the messages cam from including the names of the conferences where public messages may have originated |
+| ''CONTROL.DAT''      | QWK      | CRLF text file containing metadata about the system (BBS) where the messages came from including the names of the conferences where public messages may have originated |
 | ''DOOR.ID''          | QWK      | CRLF text file containing metadata about the software that created the QWK packet |
 | ''NETFLAGS.DAT''     | QWK      | Binary file containing a 'net status' indicator (0x00 or 0x01) for each message conference |
 | ''NEWFILES.DAT''     | QWK      | CRLF text file containing list of newly added/uploaded files on the system that created the QWK packet |
 | ''PERSONAL.NDX''     | QWK      | Binary file containing pointers to messages addressed to the user that created / downloaded the QWK packet |
-| ''000.NDX''          | QWK      | Binary file containing pointers to in the mail "inbox" for the user that created / downloaded the QWK packet |
+| ''000.NDX''          | QWK      | Binary file containing pointers to the mail "inbox" for the user that created / downloaded the QWK packet |
-| ''//x//00//y//.NDX'' | QWK      | Binary file containing pointers to messages in public message group //x//, sub-board //y//|
+| ''//x////yyy//.NDX'' | QWK      | Binary file containing pointers to messages in public message group //x//, sub-board //yyy// (Synchronet QWK conference numbering convention)|
 | ''TOREADER.EXT''     | QWKE     | CRLF text file containing metadata about the user that created / downloaded the QWK packet |
@@ Line 37: / Line 64: @@
   * All subsequent 128-byte blocks in the file are either //header blocks// or //body blocks//.
   * Each //header block// identifies a unique message in the packet.
-  * //Header blocks// are fix length (a single block only).
+  * //Header blocks// are fixed length (128 bytes).
   * Each message has only one corresponding //header block//.
   * A normal message will have at least one //body block// (at minimum).
@@ Line 72: / Line 99: @@
 | "''-''" | Public, read |
 | "''*''" | private, read but someone, but not by intended recipient |
-| "''+''" | private, read by intended recipient |
+| "''+''" | private, read by intended recipient (up for debate) |
 | "''~''" | comment to sysop, unread |
 | "''`''" | comment to sysop, read |
-| "''%''" | sender password protected, unread |
+| "''%''" | sender password protected, unread (unused by Synchronet) |
-| "''^''" | sender password protected, read |
+| "''^''" | sender password protected, read (unused by Synchronet) |
-| "''!''" | group password protected, unread |
+| "''!''" | group password protected, unread (unused by Synchronet) |
-| "''#''" | group password protected, read |
+| "''#''" | group password protected, read (unused by Synchronet) |
-| "''$''" | group password protected to all |
+| "''$''" | group password protected to all (unused by Synchronet)|
 | "''V''" | a vote message (poll, ballot, or up/down-vote) - Synchronet |
@@ Line 86: / Line 113: @@
 As you may have noticed, the QWK message header block contains some narrow limitations and bizarre inconsistencies. For example:
-  * Most fields are US-ASCII, while some are binary (little-endian) "words"
+  * Most fields are US-ASCII, while //some// are binary integers
-  * Name (to and from) are limited to 25 characters while standard FidoNet messages support up to to 35-character names
+  * Names (to and from) are limited to 25 characters while standard FidoNet messages support up to to 35-character names
-  * Message subjects are limited to 25 characters while standard FidoNet messages support up to 71-character message subjects
+  * Subjects are limited to 25 characters while standard FidoNet messages support up to 71-character subjects
-  * The message "password" field doesn't seem to have ever been used by anything, but easily **could** have been used to extend subjects to 37 characters
+  * The "password" field doesn't seem to have ever been used by anything, but easily **could** have been used to extend subjects to 37 characters
   * Message dates are limited to 2-digit years
-  * Message time is in hours and minutes only (no seconds)
+  * Message times are in hours and minutes only (no seconds)
-  * A maximum conference number of 9,999,999 is allowed in the field at offset 1 of a ''.MSG'' file, while the conference number at offset 123 is limited to just 65535.
+  * No timezone information
+  * A maximum conference number of 9,999,999 is allowed in the field at offset 1 of a ''.MSG'' file, while the conference number at offset 123 is limited to just 65,535.
   * The message number at offset 1 of a ''MESSAGES.DAT'' file is limited to 9,999,999 while the In-reply-to message number (at off 108) can hold a number up to 99,999,999.
@@ Line 105: / Line 133: @@
 For unclear reasons, new-lines sequences are normally represented by the character 0xE3 (227). In UTF-8 encoded messages, Synchronet uses ASCII 10 (LF) to represent new-lines in QWK message bodies.
-The first lines of a message's text may contain metadata about the messages (so-called "kludge lines"). Any blank lines following a set of kludge lines should be ignored.
+The first lines of a message's text may contain metadata for the message (so-called "kludge lines"). Any blank lines following a set of kludge lines should be ignored.
 == QWKE Kludge Lines ==
@@ Line 128: / Line 156: @@
 ==== HEADERS.DAT ====
-As can be seen in the definition of the QWK ''MESSAGES.DAT'' header block definition and the subsequently defined "kludge lines" to address its problems, the QWK format was not really designed to be very extensible or accommodating of newer or larger header fields. However, because the QWK packet is an archive of multiple files, adding more //standard// files is relatively trivial, hence the "new" Synchronet-defined ''HEADERS.DAT'' file.
+As can be seen in the definition of the QWK ''MESSAGES.DAT'' header block definition and the subsequently defined "kludge lines" to address its problems, the QWK format was not really designed to be very extensible or accommodating of newer or larger header fields. However, because the QWK packet is an archive of multiple files, adding more //standard// files is relatively trivial, hence the "new" Synchronet-defined ''HEADERS.DAT'' file((added to Synchronet in 2011)).
 The ''HEADERS.DAT'' is a text file (CRLF or LF-terminated lines is fine), in [[:config:ini_files|.ini file format]]. Through the use of the ''HEADERS.DAT'' file, long header field values and additional header fields may be defined and used without resulting to "kludge lines".
@@ Line 139: / Line 167: @@
 == Character Set ==
-Header field values may each be up to 1024 US-ASCII characters length. When "high ASCII" characters are included in any header field values, the IBM CP437 character set is assumed unless there is an indication that the header fields are in UTF-8 format, in which case **all** header fields with "high ASCII" characters must be in UTF-8 format. The indication of UTF-8 characters in the body text is done through the presence of either the ''X-FTN-CHRS: UTF-8 4'' header field or a ''Content-Type: text/plain; charset=utf-8'' header field.
+Header field values may each be up to 1024 US-ASCII characters in length. When "high ASCII" characters are included in any header field values, the IBM CP437 character set is assumed unless there is an indication that the header fields are in UTF-8 format, in which case **all** header fields, for a particular message, with "high ASCII" characters must be in UTF-8 format. As with FidoNet messages, the message body text and header fields must share the same encoding. If the "Utf8" ''HEADERS.DAT'' header field value for a message is set to "true", then all the headers fields and the body text of that message are to be interpreted as though they were UTF-8 encoded.
 == Example ==
@@ Line 153: / Line 181: @@
 </code>
-In this example, the //Message-ID// header field is the full RFC822-style message-ID associated with the messages in the ''MESSAGES.DAT'' a byte-offset 0xC80. The //WhenWritten/WhenImported/WhenExported// header fields provided detailed date, time, and timezone information about when the message was written, imported into the original system's message base, and exported out of the message (e.g. to a message network).
+In this example, the //Message-ID// header field is the full RFC822-style message-ID associated with the messages in the ''MESSAGES.DAT'' a byte-offset 0xC80. The //WhenWritten/Imported/Exported// header fields provide detailed date, time, and timezone information about when the message was written, imported into the original system's message base, and exported out of the original message base (e.g. to a message network).
 When a header field is present in the ''HEADERS.DAT'' file, it takes precedence over that same header field in either the ''MESSAGES.DAT'' //header block// or any //kludge lines// that may have been included in the message text (from the message's //body blocks//).
@@ Line 162: / Line 190: @@
 ^ Field          ^ Description ^
+| Utf8           | Header fields and body text use the UTF-8 character set (when ''true'') rather than CP437 (the default) |
 | Message-ID     | RFC822-style unique message identifier |
-| WhenWritten    | Date/time (in ISO-8601 format) and timezone (in SMB hexadecimal format) |
+| In-Reply-To    | RFC822-style message ID of message that this message is a reply to, if relevant |
-| WhenImported   | Date/time message was imported into the local system |
+| WhenWritten    | Date/time (in ISO-8601 format) and timezone (in hexadecimal [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbdefs.h|SMB format]]) |
-| WhenExported   | Date/time message was exported from the originating system |
+| WhenImported   | Date/time/zone message was imported into the originating system's message base |
+| WhenExported   | Date/time/zone message was exported from the originating system's message base |
 | ExportedFrom   | Details from where the message was exported: system ID, message-base, and message number |
 | Sender         | Message author/sender |
@@ Line 171: / Line 201: @@
 | SenderIpAddr   | IP address of original author's system |
 | SenderHostName | hostname of original author's system |
-| SenderProtocol | Name of protocol used by sender |
+| SenderProtocol | Name of protocol used by sender (e.g. "Telnet", "SSH", "RLogin", "SMTP, "NNTP", "HTTP", etc.) |
 | Organization   | Name of sender's organization (e.g. BBS) |
-| Reply-To       | Address to direct replies |
+| Reply-To       | The network address to direct replies to this message |
 | Subject        | Message subject |
 | To             | Message recipient name |
 | ToNetAddr      | Recipient's network address, if relevant |
 | X-FTN-AREA     | FidoNet AREA tag |
-| X-FTN-SEEN-BY  | FidoNet SEEN-BY |
+| X-FTN-SEEN-BY  | FidoNet SEEN-BY line |
-| X-FTN-PATH     | FidoNet PATH |
+| X-FTN-PATH     | FidoNet PATH line |
 | X-FTN-MSGID    | FidoNet unique message identifier |
 | X-FTN-REPLY    | FidoNet in-reply-to message identifier |
-| X-FTN-PID      | FidoNet-style program identifier of originating system system |
+| X-FTN-PID      | FidoNet-style program identifier of originating system software |
 | X-FTN-Flags    | FidoNet FLAGS kludge value |
-| X-FTN-TID      | FidoNet EchoMail Program (Tosser) Identification |
+| X-FTN-TID      | FidoNet EchoMail Program (tosser) Identification |
 | X-FTN-CHRS     | FidoNet character set (charset) identification |
 | X-FTN-Kludge   | FidoNet control paragraph (not explicitly supported with another header field) |
-| Editor         | Program (software) identifier of message editor used to create message text |
+| Editor         | Program (software) identifier of message editor used to create original message text |
 | Columns        | Number of terminal columns (if known) used by the author while composing message text |
 | Tags           | Space-separated messages tags |
@@ Line 196: / Line 226: @@
 Additional keys may be included representing not otherwise specified Internet (RFC822-style) header fields.
+==== VOTING.DAT ====
+The ''VOTING.DAT'' file is a more recent addition to the QWK packet format (introduced with Synchronet v3.17) and it contains information about message-based voting:
+  * Polls
+  * Poll-closures
+  * Ballots (votes on polls)
+  * Message up/down-votes
+Very similar to the ''HEADERS.DAT'' file, the ''VOTING.DAT'' file is a text file (with CRLF or LF-terminated lines), in [[:config:ini_files|.ini file format]].  A vote-message header block is identified in a QWK //header block// with a //Message Status// value of "''V''" (ASCII 86). The vote section of interest is located in the ''VOTING.DAT'' file by first locating the section with the corresponding //header block// byte-offset in hexadecimal (e.g. ''[f00]''), and then immediately following that section label, will be another section that identifies the //type// of vote message and this section will contain the relevant key/value pairs:
+^ Section          ^ Description   ^
+| ''[poll:<id>]''  | A posted poll with unique message-ID, //id// |
+| ''[vote:<id>]''  | A posted vote with unique message-ID, //id// |
+| ''[close:<id>]'' | The closure of previously posted poll |
+=== Polls ===
+A ''VOTING.DAT'' poll section (''[poll:<//id//>]'') may contain the following keys/values:
+| Subject          | The polling subject/question/summary |
+| Sender           | The name of the pollster |
+| Conference       | The conference number on the target system |
+| MaxVotes         | Maximum number of votes per ballot for this poll (between 1 and 16) |
+| Results          | Visibility of voting results (0=voters, 1=open, 2=closed, 3=secret) |
+| Comment//n//     | Comment line (0+) to display before available answers (optional) |
+| PollAnswer//n//  | Available answer (0+) to this poll |
+A poll is closed by a subsequent ''[close:<id>]'' vote-message-section which may occur at any later point in time (e.g. even in later QWK packets, received weeks or months later) and once discovered should mark the original poll (identified with the //In-Reply-To// key value) as unavailable for new ballot submissions.
+=== Ballots ===
+Ballot message sections are identified by a ''[vote:<//id//>]'' section. There are 2 possible types of ballots:
+  - Poll Ballots: contain a "Votes" key
+  - Message Ballots: contain an "UpVote = true" or a "DownVote = true" key/value pair
+Poll ballots are to be applied to the open-poll referenced by the "In-Reply-To" value. If the poll has been closed, then the ballot is ignored. The "Votes" key value is a bit-field (normally in hexadecimal representation with "0x" prefix), indicating which vote options the voter has selected (bit-0 = PollAnswer0, bit-1 = PollAnswer1, etc.).
+Message ballots are simply up/down-votes to be applied to a message's popularity score. The message being voted on is referenced by the "In-Reply-To" message-ID value. If the referenced message doesn't exist, the vote is ignored. The "Sender" key/value identifies the voter. Messages are never closed to new ballots, but voters should not be allowed to submit more than one ballot per message.
 ==== NDX Files ====
-The ''.NDX'' (index) files contained in QWK packets are of a notoriously bad format. These files contain byte offsets into the ''MESSAGES.DAT'' file, which is fine. But these offsets are stored as 32-bit real numbers in an obsolete "Microsoft Binary Format". Additionally, each record contain a conference number, but that conference number is only 8-bit, limiting its usefulness to only 256 possible message areas.
+The ''.NDX'' (index) files contained in QWK packets are of a notoriously bad format. These files contain byte offsets into the ''MESSAGES.DAT'' file, which is fine. But these offsets are stored as 32-bit real numbers in an obsolete "Microsoft Binary Format". Additionally, each record contains a conference number, but that conference number is only 8-bits, limiting its usefulness to only 256 possible message areas.
-Some message reader do not require or can be configured to ignore these ''.NDX'' files. There is no unique information stored in the ''.NDX'' files, so it's entirely possible for any consumer of QWK packets to be programmed to create their own useful index information on-the-fly (e.g. the first time the packet is opened).
+Some message readesr do not require or can be configured to ignore these ''.NDX'' files. There is no unique information stored in the ''.NDX'' files, so it's entirely possible for any consumer of QWK packets to be programmed to create their own useful index information on-the-fly (e.g. the first time the packet is opened).
-Synchronet does not parse or use ''.NDX'' files in any way.
+Synchronet does not parse or use ''.NDX'' files in any way, although it can generate them (user's choice).
 ==== NETFLAGS.DAT ===
@@ Line 236: / Line 302: @@
 ===== Original QWK format =====
-  * [[ftp://vert.synchro.net/main/BBS/QWKSPC11.ZIP]]
+  * [[http://vert.synchro.net/filse/Main/BBS/QWKSPC11.ZIP]]
-  * [[ftp://vert.synchro.net/modem.madness/SMMNETML/QWKINFO.ZIP]]
+  * [[http://vert.synchro.net/files/modem.madness/SMMNETML/QWKINFO.ZIP]]
-  * [[ftp://vert.synchro.net/modem.madness/SMMNETML/QWKLAY16.ZIP]]
+  * [[http://vert.synchro.net/files/modem.madness/SMMNETML/QWKLAY16.ZIP]]
-  * [[ftp://vert.synchro.net/modem.madness/SMMOLMRS/QWKDOCS.ZIP]]
+  * [[http://vert.synchro.net/files/modem.madness/SMMOLMRS/QWKDOCS.ZIP]]
-  * [[ftp://vert.synchro.net/modem.madness/SMMMAXIM/MXMS_161.ZIP]]
+  * [[http://vert.synchro.net/files/modem.madness/SMMMAXIM/MXMS_161.ZIP]]
+==== Conference Names ====
+The message area (conference) names included in the ''CONTROL.DAT'' file are only used for reader information/display purposes. The conference *numbers* are references used to identify specific message areas between the reader (or QWKnet node) and the host/hub BBS.
+=== Length ===
+According to "QWK Mail Packet File Layout" (qwklay) by Patrick Y. Lee (version 1.6 - December 19, 1992), the conference names are limited to "13 characters or less".
+According to "The Mysterious QWK-File Format" by Jeffery Foy (April 25, 1991), the conference names are limited to "10 characters or less".
+According to "QWKE Specifications 1.02" by Peter Rocca, "conference area names are no longer limited to 13 characters, they are limited to 255 characters now [in QWKE]".
 ===== Extended QWK (QWKE) Format =====
-  * [[ftp://vert.synchro.net/main/BBS/qwke.txt]].
+  * [[http://vert.synchro.net/files/main/BBS/qwke.txt]].
@@ Line 250: / Line 327: @@
   * [[network:DOVE-Net]]
   * [[:ref:|Reference Library]]
+  * [[https://www.theregister.com/2021/07/22/qwk_swatting_death/]]
 {{tag>qwk}}

Trace:

Differences

Navigation

Search

Toolbox