====== Synchronet Message Base (SMB) Specification ====== The **Synchronet Message Base** (SMB) is a binary on-disk format for storing high-volume electronic mail and conference (sub-board / forum / "echo") messages. It supports text, attachments, embedded multimedia, FidoNet kludge headers, threading, voting/polls, file metadata, and a wide range of network and gateway encodings. This page documents the SMB format itself (the on-disk layout and field semantics). For operational tools see [[util:smbutil]] (maintenance), [[util:chksmb]] (integrity check), and [[util:fixsmb]] (rebuild). For the C library that reads/writes SMB files see the [[#smblib|SMBLIB]] section. * **Current format version**: 3.10 (''0x0310''), defined as ''SMB_VERSION'' in [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smblib.c|src/smblib/smblib.c]] * **Original spec**: v1.21 (1993), preserved at [[https://synchro.net/docs/smb.html]] Since the original spec was published, the format has been extended significantly. This page reflects the current behavior, modernized from the original ''/sbbs/docs/smb.html''. Where the original normative wording remains accurate, it is preserved. ===== Introduction ===== ==== What is SMB? ==== SMB is the on-disk format used to store messages — both private mail and public conference posts (sub-boards, forums, conferences, SIGs, echoes). A single message base is one logical "mailbag" or "conference"; a Synchronet system has one mail base plus one base per sub-board. ==== Why SMB? ==== SMB is designed for **high-volume** message bases under multi-user concurrent access. It supports fast incremental imports (at echomail-tosser rates), random retrieval by various keys, multiple storage tradeoffs (Self-packing / Fast / Hyper allocation), flexible content (text, attachments, multimedia), and extensibility for new field types and network encodings. ==== Implementations ==== The format is open. Third-party programs (message readers, mail tossers, gateways, archive tools) may read and write SMB bases using either the published spec (this page) or the reference C library [[#smblib|SMBLIB]] (LGPL). ===== Definitions ===== ==== Notation ==== * **Control characters** — Written as ''^X'' (caret + letter) or ''ctrl-X''. Case-insensitive (''^z'' = ''^Z''). ''^@'' (ASCII 0) is written ''NULL'' or ''0''. * **Hexadecimal** — Numbers are prefixed ''0x'' or ''\x'' or suffixed ''h''. Case-insensitive (''0xff'' = ''0xFF''). * **Bit values** — In C notation as ''(1< typedef struct { /* Time with time-zone */ uint16_t year; uint32_t time; /* Month/Day/Hour/Minute/Second OR (legacy) time_t */ int16_t zone; /* Time zone */ } when_t; The ''time'' field is **dual-format**, distinguished by its **upper 6 bits**: * **Upper 6 bits non-zero** → //legacy// 32-bit ''time_t'' (Unix epoch seconds in **local time** at the writer's site). Used by all SMB messages written before SMBLIB v3.10, and still used today for ''when_imported''. * **Upper 6 bits zero** → //encoded wallclock//: Month/Day/Hour/Minute/Second packed into the low 26 bits, with the year stored separately in the ''year'' field. Used by ''when_written'' since SMBLIB v3.10. The wallclock-encoded layout is: ^ Field ^ Width ^ Bit position ^ Macros ^ | Month | 4 bits | bits 22-25 | ''SMB_DATE_MON_BITWIDTH'' / ''SMB_DATE_MON_BITPOS'' | | Day | 5 bits | bits 17-21 | ''SMB_DATE_DAY_BITWIDTH'' / ''SMB_DATE_DAY_BITPOS'' | | Hour | 5 bits | bits 12-16 | ''SMB_DATE_HR_BITWIDTH'' / ''SMB_DATE_HR_BITPOS'' | | Minute | 6 bits | bits 6-11 | ''SMB_DATE_MIN_BITWIDTH'' / ''SMB_DATE_MIN_BITPOS'' | | Second | 6 bits | bits 0-5 | ''SMB_DATE_SEC_BITWIDTH'' / ''SMB_DATE_SEC_BITPOS'' | This change was introduced in commit ''[[https://gitlab.synchro.net/main/sbbs/-/commit/445394f9fc31d3d40261737c101ff39fe6ba01d0|445394f9]]'' (2024-12-20), which incremented ''SMB_VERSION'' to ''0x0310''. The motivation: with the old time_t encoding, **changing the system or OS time zone changed the displayed dates of all stored messages** (issue #845). The new wallclock encoding stores what the message author saw on their clock when writing, so the displayed date never shifts with subsequent timezone changes. As a side benefit it sidesteps the 32-bit ''time_t'' Y2038 / Y2106 problems — at least for ''when_written''. The ''year'' field repurposes 16 bits previously taken by the upper half of ''netattr'' (which was reduced from ''uint32_t'' to ''uint16_t'' in the same commit). File-base entries still use ''time_t'' for ''when_written'' — they were not converted, since the field is not used for much in file metadata. ''when_t.zone'' encoding: * In the range ''-720..+720'' — minutes east (positive) or west (negative) of UT. * Outside ''±720'' — zone code with bit fields: * ''0x1000'' — Non-US zone, east of UT * ''0x2000'' — Non-US zone, west of UT * ''0x4000'' — U.S. zone (Uniform Time Act) * ''0x8000'' — Daylight savings active * Lower 12 bits — minutes east/west of UT (without DST adjustment). * A value of ''0'' for the entire timestamp is invalid (un-initialized). Common U.S. zone codes: ^ Code ^ Hex (Std) ^ Hex (DST) ^ Offset ^ | AST | ''0x40F0'' | ''0xC0F0'' (ADT) | -04:00 / -03:00 | | EST | ''0x412C'' | ''0xC12C'' (EDT) | -05:00 / -04:00 | | CST | ''0x4168'' | ''0xC168'' (CDT) | -06:00 / -05:00 | | MST | ''0x41A4'' | ''0xC1A4'' (MDT) | -07:00 / -06:00 | | PST | ''0x41E0'' | ''0xC1E0'' (PDT) | -08:00 / -07:00 | | YST | ''0x421C'' | ''0xC21C'' (YDT) | -09:00 / -08:00 | | HST | ''0x4258'' | ''0xC258'' (HDT) | -10:00 / -09:00 | | BST | ''0x4294'' | ''0xC294'' (BDT) | -11:00 / -10:00 | A range of non-standard zone codes for international cities is also defined; see [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbdefs.h|smbdefs.h]] for the complete table. ''fidoaddr_t'' — FidoNet address (zone:net/node.point): typedef struct { uint16_t zone; uint16_t net; uint16_t node; uint16_t point; } fidoaddr_t; All on-disk SMB structures are written with ''#pragma pack(push,1)'' / ''pack(pop)'' bracketing them — they are byte-packed without alignment padding. ===== File Formats ===== Each message base is stored as a set of binary files sharing a common base name (max 8 chars under DOS, longer permitted on modern systems) but differing by file extension. The first character of the extension is always ''S'' (for SMB). ^ Extension ^ Purpose ^ Re-creatable? ^ | ''.sid'' | Index — fixed-length per-message records for fast lookup | Yes (from ''.shd'') | | ''.shd'' | Header — variable-length per-message records (256-byte blocks) | **No** — primary data | | ''.sha'' | Header block allocation table | Yes (from ''.shd'') | | ''.sdt'' | Data — message bodies and attachments (256-byte blocks) | **No** — primary data | | ''.sda'' | Data block allocation table | Yes (from ''.shd'') | | ''.sch'' | Optional CRC history for duplicate-message detection | Yes | | ''.hash'' | Hash file with multiple digests for duplicate detection | Yes | | ''.ini'' | Auxiliary configuration (created at base creation; minimum: ''Created'' timestamp) | — | | ''.lock'' | Whole-base exclusive lock (transient — created by ''[[#whole-base_lock|smb_lock()]]'', removed by ''smb_unlock()''; presence means another process is performing maintenance/repair) | — | Allocation files (''.sda'', ''.sha'') are not used when the base uses **Hyper Allocation** storage method. ==== Index File (.sid) ==== Fixed-length records, **20 bytes** each (''SIZEOF_SMB_IDXREC_T''), one per message. The first 6 bytes are a union discriminated by the message ''type'' in the corresponding header: typedef struct { /* Index record */ union { struct { /* msg.type != BALLOT/FILE (normal message) */ uint16_t to; /* 16-bit CRC of recipient name (lowercase) or user # */ uint16_t from; /* 16-bit CRC of sender name (lowercase) or user # */ uint16_t subj; /* 16-bit CRC of subject (lowercase, w/o "RE:") */ }; struct { /* msg.type == BALLOT */ uint16_t votes; /* votes value */ uint32_t remsg; /* number of message this vote responds to */ }; struct { /* msg.type == FILE */ uint32_t size; uint16_t size_ext; /* additional 16 bits of file size */ }; }; uint16_t attr; /* attributes (MSG_PRIVATE, MSG_READ, etc.; mirrored in msghdr) */ uint32_t offset; /* byte-offset of msghdr in header file */ uint32_t number; /* message number (1-based) */ uint32_t time; /* time/date message was imported/posted (32-bit time_t) */ } idxrec_t; For the **mail** base, ''to'' and ''from'' contain user numbers (not CRCs). For sub-boards, they contain 16-bit CRCs of the lowercased names, which speeds searches but allows aliasing. The message ''type'' is held in the corresponding header record (''msghdr_t.type''); ''enum smb_msg_type'' values: ''SMB_MSG_TYPE_NORMAL'' (0), ''SMB_MSG_TYPE_POLL'', ''SMB_MSG_TYPE_BALLOT'', ''SMB_MSG_TYPE_POLL_CLOSURE'', ''SMB_MSG_TYPE_FILE''. The index file is fully re-creatable from the header file by walking the headers and emitting one record per message — this is what [[util:fixsmb]] does. ==== Header File (.shd) ==== The .shd file begins with an SMB header followed by a status header, then variable-length message header records aligned on 256-byte boundaries. === SMB header (file root) === typedef struct { uchar smbhdr_id[4]; /* "SMB\x1A" (SMB_HEADER_ID) */ uint16_t version; /* SMB format version (e.g. 0x0310) */ uint16_t length; /* Length of this header in bytes */ } smbhdr_t; === Status header (follows SMB header) === The status header doubles as the file-base status header — fields with parallel meanings between message and file bases use a C union: typedef struct { /* Message/File base status header */ union { uint32_t last_msg; /* last message number */ uint32_t last_file; /* last file number */ }; union { uint32_t total_msgs; /* total messages */ uint32_t total_files; /* total files */ }; uint32_t header_offset; /* byte offset to first header record */ uint32_t max_crcs; /* maximum number of CRCs to keep in history */ union { uint32_t max_msgs; /* maximum messages to keep */ uint32_t max_files; /* maximum files to keep */ }; uint16_t max_age; /* maximum age (days), 0 = unlimited */ uint16_t attr; /* base-wide SMB_* attribute flags */ } smbstatus_t; Status header ''attr'' (base-wide flags): ^ Flag ^ Hex ^ Meaning ^ | ''SMB_EMAIL'' | ''0x01'' | Mail base — ''to''/''from'' indexes hold user numbers, not CRCs | | ''SMB_HYPERALLOC'' | ''0x02'' | Hyper Allocation method (no .sda/.sha files) | | ''SMB_NOHASH'' | ''0x04'' | Don't calculate or store hashes | | ''SMB_FILE_DIRECTORY''| ''0x08'' | Storage for a file area (not message base) | Storage method values used elsewhere (passed to functions, not stored as status flags) are ''SMB_SELFPACK'' = 0, ''SMB_FASTALLOC'' = 1, ''SMB_HYPERALLOC'' = 2. === Message header record === Each message header begins with a fixed-length ''msghdr_t'' structure, followed by an array of //data field records// (''dfield_t'') for the body, then a sequence of variable-length //header field records// (each a ''hfield_t'' followed by its data bytes). The whole record is padded to a multiple of 256 bytes (''SHD_BLOCK_LEN''). typedef struct { /* Message/File header */ /* 00 */ uchar msghdr_id[4]; /* "SHD\x1A" (SHD_HEADER_ID) */ /* 04 */ uint16_t type; /* enum smb_msg_type */ /* 06 */ uint16_t version; /* version of this header format */ /* 08 */ uint16_t length; /* total length of fixed record + all fields */ /* 0a */ uint16_t attr; /* primary attributes (duped in idx) */ /* 0c */ uint32_t auxattr; /* auxiliary attributes */ /* 10 */ uint16_t netattr; /* network attributes (was uint32_t before v3.10) */ /* 12 */ when_t when_written; /* date/time/zone written (8 bytes; v3.10 wallclock or legacy time_t) */ /* 1a */ struct { uint32_t time; /* legacy 32-bit time_t (always) */ int16_t zone; } when_imported; /* date/time/zone imported (6 bytes; not a when_t) */ /* 20 */ uint32_t number; /* message number */ /* 24 */ uint32_t thread_back; /* aka thread_orig — original message in thread */ /* 28 */ uint32_t thread_next; /* next reply in thread */ /* 2c */ uint32_t thread_first; /* first reply to this message */ /* 30 */ uint16_t delivery_attempts; /* SMTP delivery attempts */ /* 32 */ int16_t votes; /* poll: max votes per ballot; ballot: response value */ /* 34 */ uint32_t thread_id; /* number of original in thread (or 0 if unknown) */ /* 38 */ union { struct { /* for messages */ uint8_t priority; /* enum smb_priority */ }; struct { /* for files */ uint32_t times_downloaded; uint32_t last_downloaded; }; }; /* 40 */ uint32_t offset; /* offset for buffer into data file (0 or mod 256) */ /* 44 */ uint16_t total_dfields; /* total number of dfield records following */ } msghdr_t; Notes: * ''when_imported'' is **not** a ''when_t''. It uses the pre-v3.10 inline layout (''uint32_t time'' + ''int16_t zone''), with ''time'' always interpreted as a 32-bit Unix ''time_t''. This was retained in the v3.10 commit (''[[https://gitlab.synchro.net/main/sbbs/-/commit/445394f9fc31d3d40261737c101ff39fe6ba01d0|445394f9]]'') for backward compatibility — the same commit converted ''when_written'' to the new ''when_t'' (with ''year'') and reduced ''netattr'' from 32-bit to 16-bit to make room. * ''thread_back'' is also exposed under the legacy name ''thread_orig'' via ''#define''. * Messages with ''type == SMB_MSG_TYPE_FILE'' use the file-base variant of the union (download counters); other types use the ''priority'' variant. ==== Header Block Allocation (.sha) ==== For each 256-byte block of the .shd file, the .sha file holds **one byte** of allocation state: * ''0x00'' — block is free (deleted/unused) * non-zero — block is in use (the library writes ''0x01'') The .sha file enables blocks freed by message deletion to be reused for new messages without "holes" in the file. Re-creatable from .shd. Not used in **Hyper Allocation** mode. ==== Message Data (.sdt) ==== The .sdt file holds message body text, attachments, and embedded media. Like .shd, it uses 256-byte blocks (''SDT_BLOCK_LEN''). Each //data field record// in a message header points to a contiguous run of blocks here, with type, offset, and length. Message text is stored as one or more text records (each prefixed by a translation-string and followed by the body bytes), each in its own data field. ==== Data Block Allocation (.sda) ==== For each 256-byte block of the .sdt file, the .sda file holds a **16-bit** ''uint16_t'' reference count: * ''0x0000'' — block is free (deleted/unused) * non-zero — number of messages currently referencing this block (data blocks may be shared between message-copies, e.g. via forwarding, hence the refcount rather than a simple in-use flag) When a message is freed, the refcount is decremented and the block is freed only when it reaches zero. ''SMB_ALL_REFS'' (= 0) passed to ''smb_freemsgdat()'' forces all references to be cleared regardless of count. Re-creatable from .shd. Not used in Hyper Allocation mode. ==== CRC History (.sch) ==== Optional file holding 32-bit CRC-32 values of recently fingerprinted content, capped to ''max_crcs'' entries (set in the status header). Maintained by ''smb_addcrc()''; new entries push old ones out FIFO when the cap is reached. In modern Synchronet, the ''.sch'' file is **not** the primary duplicate-detection mechanism. ''smb_addmsg()'' (the canonical entry point in [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbadd.c|smbadd.c]]) checks the ''.hash'' file via ''smb_findhash()'' and skips dupe-checking entirely for bases with ''SMB_EMAIL'', ''SMB_NOHASH'', or ''SMB_FILE_DIRECTORY'' set. The ''.sch'' file is still actively maintained by ''email.cpp'' for the mail base and by some legacy import paths, but ''.hash'' is the mechanism for sub-boards and file bases. ==== Whole-base Lock ==== A ''.lock'' file is used by ''smb_lock()'' as a //cooperative// exclusive lock on the entire base. It is created with ''O_CREAT | O_EXCL'' (atomic create-or-fail), so concurrent attempts to create it fail until the holder deletes it via ''smb_unlock()''. ''smb_islocked()'' tests for its presence. This is **separate from** the in-process file locking used during normal reads/writes (''smb_locksmbhdr()'' / ''smb_lockmsghdr()'' apply ''fcntl''-style record locks on the ''.shd'' file itself). The ''.lock'' file is intended for offline maintenance/repair operations — typically by the sysop running ''[[util:smbutil]]'', ''[[util:chksmb]]'', or ''[[util:fixsmb]]'' — that need to exclude the BBS and other tools for the duration. If a process holding the lock crashes or is killed without removing the lock file, the lock will persist indefinitely. Manually removing a stale ''.lock'' file is safe **only if** no other process is actually operating on the base. ==== Hash File (.hash) ==== Modern duplicate-detection file containing one ''hash_t'' record (64 bytes, ''SIZEOF_SMB_HASH_T'') per fingerprinted source. Each record carries up to four digests (CRC-16, CRC-32, MD5, SHA-1; see [[#hash_flags|hash flags]]) and a ''source'' field identifying what was hashed (''SMB_HASH_SOURCE_BODY'', ''SMB_HASH_SOURCE_MSG_ID'', ''SMB_HASH_SOURCE_FTN_ID'', ''SMB_HASH_SOURCE_SUBJECT''). For sub-boards, ''smb_addmsg()'' computes the dupe-source set ''SMB_HASH_SOURCE_DUPE'' (''BODY'' | ''MSG_ID'' | ''FTN_ID'') and rejects a new message if any of those hashes match an existing entry. For file bases the relevant hashes (configured per directory) gate uploads. Re-creatable via ''smbutil''. ==== Storage Methods ==== ^ Method ^ Constant ^ .sda/.sha used? ^ Speed ^ Disk usage ^ Notes ^ | Self-packing | ''SMB_SELFPACK'' (0) | Yes | Slowest imports | Smallest | Reuses freed blocks; no periodic packing needed for typical usage | | Fast Allocation | ''SMB_FASTALLOC'' (1) | Yes | Faster imports | Grows over time; needs ''[[util:smbutil]] p'' periodically | | | Hyper Allocation | ''SMB_HYPERALLOC'' ((1<<1)) | No | Fastest imports | Grows; needs ''[[util:smbutil]] p'' periodically; no allocation files | Selectable for sub-boards via the ''SUB_HYPER'' [[config:message_areas#sub-board_advanced_options|toggle]]. The library accepts it for any base, but the BBS's ''smb_storage_mode()'' (in ''src/sbbs3/load_cfg.c'') never selects Hyper for the mail base — mail uses Self-packing or Fast (''SM_FASTMAIL'' system flag). | ===== Message Bases vs. File Bases ===== The same SMB format and library serve two distinct purposes in modern Synchronet: * **Message bases** — store messages: e-mail (the system mail base) and conference posts (one base per sub-board). * **File bases** — store //file records// (filename, description, uploader, hashes) for the file transfer section. The actual file contents live elsewhere on disk; the SMB base is the metadata catalog. Each "message" in a file base represents one file. File bases were converted to use the SMB format in **Synchronet v3.19** (see [[history:newfilebase]] for the migration details). Prior to v3.19, file bases used a separate format (''.ixb'' / ''.dat'' / ''.exb'' / ''.dab''). The conversion was automatic via the one-time ''upgrade_to_v319'' utility. ==== Storage Locations ==== ^ Type ^ Location ^ Notes ^ | Mail base | ''[[dir:data]]/mail.s*'' | One per system; ''SMB_EMAIL'' status flag set | | Sub-board (message) bases | ''[[dir:data]]/subs/.s*'' | One per sub-board, named by the sub-board's [[config:message_areas|internal code]] | | File bases | ''[[dir:data]]/dirs/.s*'' | One per file directory, named by the directory's [[config:file_areas|internal code]]; ''SMB_FILE_DIRECTORY'' status flag set | ==== Discriminator ==== The status header ''attr'' field tells client code which kind of base it is reading: ^ Flag ^ Hex ^ Meaning ^ | ''SMB_EMAIL'' | ''0x01'' | This is the mail base (''to''/''from'' indexes hold user numbers, not name CRCs) | | ''SMB_FILE_DIRECTORY''| ''0x08'' | This is a **file base**, not a message base | A normal sub-board base has neither flag set. ==== What Differs ==== Although both share the same on-disk file set (''.shd'' / ''.sid'' / ''.sdt'' / ''.sda'' / ''.sha'' / ''.hash'' / ''.ini''), the //meaning// of records differs: ^ Aspect ^ Message base ^ File base ^ | Index record | ''idxrec_t'' (20 bytes; ''SIZEOF_SMB_IDXREC_T'') — ''to''/''from''/''subj'' CRCs (or user numbers for mail; or ballot/file variants via union), ''attr'', ''offset'', ''number'', ''time'' | ''fileidxrec_t'' (128 bytes; ''SIZEOF_SMB_FILEIDXREC_T'') — embedded ''idxrec_t'' (with the //file// union variant: ''size''/''size_ext''), plus ''name[SMB_FILEIDX_NAMELEN+1]'' (64+1 chars indexed) and a ''struct hash_info'' (CRC16/CRC32/MD5/SHA1) | | Each "record" represents | One message | One file (filename, description, uploader, hashes, etc.) | | Header fields (typical) | ''SENDER'', ''RECIPIENT'', ''SUBJECT'', ''SMB_SUMMARY'', ''SMB_AUTHOR'', RFC822/FidoNet kludges, … | ''SMB_FILEUPLOADER'' (= ''SENDER''), ''SMB_FILENAME'' (= ''SUBJECT''), ''SMB_FILEDESC'' (= ''SMB_SUMMARY''), ''SMB_TAGS''; optionally ''RECIPIENTLIST'' for [[#user-to-user_file_transfers|user-to-user file transfers]] | | Body data fields | ''TEXT_BODY'' (and optional ''TEXT_TAIL'') containing the message text | Extended (multi-line) file description text | | Hash file (''.hash'') | Used (when enabled) for duplicate **message** detection (typically by message body) | Used (when enabled) for duplicate **file** detection — multiple algorithms (CRC16/CRC32/MD5/SHA1) computed at upload time | | Recipient-based lookup | Via ''to'' index (CRC of name) | Files have no recipient //in the index//, but a file may have a ''RECIPIENTLIST'' header field naming destination users (see [[#user-to-user_file_transfers|user-to-user transfers]] below) | ==== Hash Flags ==== The per-record hash structure (''hash_t'' / ''struct hash_data'') uses these flags to indicate which algorithms have been computed and stored: ^ Flag ^ Hex ^ Hash ^ | ''SMB_HASH_CRC16'' | ''0x01'' | CRC-16 | | ''SMB_HASH_CRC32'' | ''0x02'' | CRC-32 | | ''SMB_HASH_MD5'' | ''0x04'' | MD5 | | ''SMB_HASH_SHA1'' | ''0x08'' | SHA-1 | By default Synchronet computes all four during upload. Hashing can be disabled per-directory in [[util:scfg|SCFG]] → File Areas → //Toggle Options// (e.g. for very large files or for performance). ==== Field Aliases ==== Modern source uses the following compile-time aliases when writing to or reading from a file base, mapping file-base concepts onto the same header field type codes used by message bases: ^ File-base alias ^ Equals ^ Hex ^ Meaning ^ | ''SMB_FILEUPLOADER'' | ''SENDER'' | ''0x00'' | The user who uploaded the file | | ''SMB_FILENAME'' | ''SUBJECT'' | ''0x60'' | The filename | | ''SMB_FILEDESC'' | ''SMB_SUMMARY'' | ''0x61'' | The single-line description ("file summary") | Long extended descriptions are stored as ''TEXT_BODY'' data fields, the same way long message bodies are stored in message bases. ==== Tools and APIs ==== All SMB-aware tools work on either kind of base: * [[util:smbutil]] — operates on any ''.shd'' file (message or file base) * [[util:chksmb]] — integrity check * [[util:fixsmb]] — index rebuild JavaScript APIs: * ''MsgBase'' class — for message bases * ''FileBase'' class — for file bases (added in v3.19; see [[module:fileman|fileman.js]] / ''addfiles.js'' / ''postfile.js'' / ''filelist.js'' / ''delfiles.js'' / ''dupefind.js'') ==== User-to-User File Transfers ==== A file uploaded into a sysop-designated //user-to-user// directory may be addressed to one or more specific destination users (rather than the public file base). This is implemented in the file base by attaching a ''RECIPIENTLIST'' header field to the file's record — a comma-separated list of destination **user numbers**. * The in-BBS upload path stores the recipients via ''smb_hfield_str(&f, RECIPIENTLIST, ...)'' (see ''src/sbbs3/upload.cpp''). * The ''FileBase'' JavaScript class exposes this as the ''to_list'' property on a file object: a comma-separated list of recipient user numbers (see ''src/sbbs3/js_filebase.cpp''). * Recipients are addressed by **user number**, not name — analogous to how the **mail** message base stores user numbers (rather than name CRCs) in its index. * The destination users retrieve the file with the ''/D'' command at the transfer menu; the uploader sends with ''/U''. A file base directory is configured as user-to-user by enabling the appropriate //Toggle Option// in [[util:scfg|SCFG]] → File Areas. The legacy ''xfer.ixt'' file (pre-v3.19) tracked pending user-to-user transfers; existing entries in that file are not migrated by ''upgrade_to_v319'' — see [[history:newfilebase#user-to-user_file_transfers]] for migration notes. ===== Header Field Types ===== A message header consists of a fixed-length prefix followed by zero or more //header field records//. Each field has the form: typedef struct { /* Header field */ uint16_t type; /* field type code */ uint16_t length; /* length of the field data following this struct */ } hfield_t; /* the struct is immediately followed in the file by 'length' bytes of data */ Header fields hold sender/recipient information, subject, network addresses, kludge lines, and many other attributes. Field type codes are 16-bit and grouped by range. See [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbdefs.h|smbdefs.h]] for the canonical list of type codes and their on-disk numeric values. ^ Range ^ Category ^ Examples ^ | ''0x00''-''0x0e'' | Sender | ''SENDER'', ''SENDERAGENT'', ''SENDERNETTYPE'', ''SENDERNETADDR'', ''SENDEREXT'', ''SENDERPOS'', ''SENDERORG'', ''SENDERIPADDR'', ''SENDERHOSTNAME'', ''SENDERPROTOCOL'', ''SENDERPORT'', ''SENDERUSERID'', ''SENDERTIME'', ''SENDERSERVER'' | | ''0x10''-''0x16'' | Author | ''SMB_AUTHOR'', ''SMB_AUTHOR_ORG'' | | ''0x30''-''0x37'' | Recipient | ''RECIPIENT'', ''RECIPIENTAGENT'', ''RECIPIENTNETTYPE'', ''RECIPIENTNETADDR'', ''RECIPIENTEXT'', ''RECIPIENTPOS'', ''RECIPIENTORG'', ''RECIPIENTLIST'' | | ''0x48'' | Forwarding | ''FORWARDED'' | | ''0x60''-''0x6a'' | Subject / metadata | ''SUBJECT'', ''SMB_SUMMARY'', ''SMB_COMMENT'', ''SMB_CARBONCOPY'', ''SMB_GROUP'', ''SMB_EXPIRATION'', ''SMB_COST'', ''SMB_EDITOR'', ''SMB_TAGS'', ''SMB_COLUMNS'' | | ''0x70''-''0x72'' | File attach | ''FILEATTACH'', ''DESTFILE'', ''FILEATTACHLIST'' | | ''0xa0''-''0xaa'' | FidoNet kludges | ''FIDOCTRL'' (generic), ''FIDOAREA'', ''FIDOSEENBY'', ''FIDOPATH'', ''FIDOMSGID'', ''FIDOREPLYID'', ''FIDOPID'', ''FIDOFLAGS'', ''FIDOTID'', ''FIDOCHARSET'', ''FIDOBBSID'' | | ''0xb0''-''0xb8'' | RFC822 / Internet | ''RFC822HEADER'', ''RFC822MSGID'', ''RFC822REPLYID'', ''RFC822TO'', ''RFC822FROM'', ''RFC822REPLYTO'', ''RFC822CC'', ''RFC822ORG'', ''RFC822SUBJECT'' | | ''0xe0'' | Polls | ''SMB_POLL_ANSWER'' (poll answer; the subject is the question) | | ''0xf1''-''0xff'' | Other / unused | ''UNKNOWN'', ''UNKNOWNASCII'', ''UNUSED'' | The file-base entry types reuse some of the above with aliases: ''SMB_FILENAME'' = ''SUBJECT'', ''SMB_FILEDESC'' = ''SMB_SUMMARY'', ''SMB_FILEUPLOADER'' = ''SENDER''. ==== RECIPIENT and related ==== * ''RECIPIENT'' — single name or full network address of the message recipient (US-ASCII or UTF-8). **Every message must have a single ''RECIPIENT'' header field.** * ''RECIPIENTLIST'' — original comma-separated list of all recipients (excluding CCs). Optional. * ''RFC822TO'' — MIME-encoded version of ''RECIPIENTLIST''. Should not exist without ''RECIPIENTLIST''. ===== Data Field Types ===== A //data field record// (''dfield_t'') in a message header points to a region of the .sdt file: typedef struct { /* Data field */ uint16_t type; /* data field type code */ uint32_t offset; /* byte offset within .sdt */ uint32_t length; /* length of the data region */ } dfield_t; Currently defined data field types (''dfield_t.type''): ^ Type ^ Hex ^ Holds ^ | ''TEXT_BODY'' | ''0x00'' | The plain message body text (usually one per message; multiples are concatenated when displayed) | | ''TEXT_TAIL'' | ''0x02'' | Tear-line / origin / SEEN-BY block (FidoNet) | File attachments are referenced via the [[#header_field_types|header field]] ''FILEATTACH'', not via a dedicated data field type. ===== Message Attributes ===== The 16-bit ''attr'' field of the ''idxrec_t'' / ''msghdr_t'' carries the primary message-level flags. The ''auxattr'' field adds extended (auxiliary) flags, and ''netattr'' adds network-related flags. Primary attribute flags (''attr''): ^ Flag ^ Hex ^ Meaning ^ | ''MSG_PRIVATE'' | ''0x0001'' | Private message | | ''MSG_READ'' | ''0x0002'' | Recipient has read | | ''MSG_PERMANENT'' | ''0x0004'' | Don't auto-delete by age/count | | ''MSG_LOCKED'' | ''0x0008'' | //(deprecated, never used)// | | ''MSG_DELETE'' | ''0x0010'' | Marked for deletion | | ''MSG_ANONYMOUS'' | ''0x0020'' | Author is anonymous | | ''MSG_KILLREAD'' | ''0x0040'' | Delete after recipient reads | | ''MSG_MODERATED'' | ''0x0080'' | Awaiting moderator validation | | ''MSG_VALIDATED'' | ''0x0100'' | Sysop-validated | | ''MSG_REPLIED'' | ''0x0200'' | Recipient has replied | | ''MSG_NOREPLY'' | ''0x0400'' | No replies (or bounces) should be sent to the sender | | ''MSG_UPVOTE'' | ''0x0800'' | Upvote (poll vote) | | ''MSG_DOWNVOTE'' | ''0x1000'' | Downvote (poll vote) | | ''MSG_POLL'' | ''0x2000'' | Message is a poll | | ''MSG_SPAM'' | ''0x4000'' | Flagged as SPAM | | ''MSG_FILE'' | ''0x8000'' | This is a file (file-base entry, not a message) | | ''MSG_VOTE'' | ''0x1800'' | Composite: ''MSG_UPVOTE \| MSG_DOWNVOTE'' | | ''MSG_POLL_CLOSURE'' | ''0x3800'' | Composite: ''MSG_POLL \| MSG_VOTE'' | Auxiliary attribute flags (''auxattr''): ^ Flag ^ Hex ^ Meaning ^ | ''MSG_FILEREQUEST'' | ''0x00000001'' | FidoNet file request | | ''MSG_FILEATTACH'' | ''0x00000002'' | Carries attached file(s) (paths/names in subject) | | ''MSG_MIMEATTACH'' | ''0x00000004'' | One or more MIME-embedded attachments | | ''MSG_KILLFILE'' | ''0x00000008'' | Delete file(s) when sent | | ''MSG_RECEIPTREQ'' | ''0x00000010'' | Return receipt requested | | ''MSG_CONFIRMREQ'' | ''0x00000020'' | Confirmation receipt requested | | ''MSG_NODISP'' | ''0x00000040'' | Message may not be displayed to user | | ''MSG_FIXED_FORMAT'' | ''0x00000080'' | Preformatted message body text | | ''MSG_HFIELDS_UTF8'' | ''0x00002000'' | Message header fields are UTF-8 encoded | | ''POLL_CLOSED'' | ''0x01000000'' | Closed to voting | Network attribute flags (''netattr''): ^ Flag ^ Hex ^ Meaning ^ | ''NETMSG_LOCAL'' | ''0x0001'' | Locally created | | ''NETMSG_INTRANSIT'' | ''0x0002'' | In transit (for batch processing) | | ''NETMSG_SENT'' | ''0x0004'' | Sent to remote | | ''NETMSG_KILLSENT'' | ''0x0008'' | Kill when sent | | ''NETMSG_HOLD'' | ''0x0020'' | Hold for pick-up | | ''NETMSG_CRASH'' | ''0x0040'' | High priority (crash mail) | | ''NETMSG_IMMEDIATE'' | ''0x0080'' | Send immediately, ignore restrictions | | ''NETMSG_DIRECT'' | ''0x0100'' | Send directly to destination | For the complete and current set, see [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbdefs.h|smbdefs.h]]. ===== Translation Types ===== A 16-bit ''xlat'' code identifies an encoding/compression transform applied to data. Multiple translations may be chained. Currently defined translation types (''enum smb_xlat_type''): ^ Code ^ Name ^ Meaning ^ | ''0'' | ''XLAT_NONE'' | No translation; also the terminator of an xlat array | | ''9'' | ''XLAT_LZH'' | LHarc (LHA) Dynamic Huffman coding | Apply translations in **storage order** to encode; reverse the order to decode. ===== Agent Types ===== The ''SENDERAGENT'' and ''RECIPIENTAGENT'' header fields identify the //software agent// type at each end (''enum smb_agent_type''): ^ Code ^ Name ^ Meaning ^ | ''0'' | ''AGENT_PERSON'' | A human user | | ''1'' | ''AGENT_PROCESS'' | Unknown automated process | | ''2'' | ''AGENT_SMBUTIL'' | Imported via Synchronet [[util:smbutil|SMBUTIL]] | | ''3'' | ''AGENT_SMTPSYSMSG'' | Synchronet [[server:mail|SMTP server]] system message | ===== Network Types ===== ''SENDERNETTYPE'' / ''RECIPIENTNETTYPE'' identify which network's address format is in the corresponding ''*NETADDR'' field (''enum smb_net_type''): ^ Code ^ Name ^ Network ^ | ''0'' | ''NET_NONE'' | Local message (no network) | | ''1'' | ''NET_UNKNOWN'' | Unknown network type | | ''2'' | ''NET_FIDO'' | FidoNet address, ''faddr_t'' format (4D) | | ''3'' | ''NET_POSTLINK'' | Imported with UTI driver | | ''4'' | ''NET_QWK'' | QWK networked message | | ''5'' | ''NET_INTERNET'' | Internet e-mail, netnews, etc. | ===== Message Storage Pseudo-Code ===== To **add** a new message to a base (the ''smb_addmsg()'' flow in [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbadd.c|smbadd.c]]): - Auto-create the base if the ''.shd'' file is empty (''smb_create()''). - Lock the SMB header (exclusive). - Read the ''smbstatus_t''. - Compute the new message's serial number as ''status.last_msg + 1''. - **Unless** the base has ''SMB_EMAIL'', ''SMB_NOHASH'', or ''SMB_FILE_DIRECTORY'' set, compute the dupe-source hashes (''smb_msghashes()'' for ''SMB_HASH_SOURCE_DUPE'' = ''BODY'' | ''MSG_ID'' | ''FTN_ID'') and check against the ''.hash'' file (''smb_findhash()''). If a match is found, abort with ''SMB_DUPE_MSG''. - Apply translations to the body in **storage order** (the ''xlat[]'' chain — only ''XLAT_LZH'' is currently implemented). - Allocate space in the ''.sdt'' file for each data field (text body, tail) according to the storage method: * **Hyper Allocation** (''smb_hallocdat()'') — append at end of file; no .sda update. * **Fast Allocation** (''smb_fallocdat()'') — append at end; record refcount in ''.sda''. * **Self-packing** (''smb_allocdat()'') — search ''.sda'' for a contiguous free run of the required size; allocate there. Append if no fit. - Build the ''msghdr_t'' with the ''dfield[]'' entries pointing into the .sdt regions and the variable-length header field records. - Allocate a header-aligned region in ''.shd'' (''smb_hallochdr()'' / ''smb_fallochdr()'' / ''smb_allochdr()''). - Write the header (''smb_putmsg()'') — this writes the ''msghdr_t'' + dfields + header fields, then writes the corresponding ''idxrec_t'' to ''.sid''. - Increment ''status.last_msg'' and ''status.total_msgs''; write status (''smb_putstatus()''). - Add the new message's hashes to ''.hash'' (when applicable). - Unlock the SMB header. ===== Message Retrieval Pseudo-Code ===== To **read** a message by index entry: - Open the ''.shd'' file (shared lock). - From the ''idxrec_t'', take the ''offset'' and seek into ''.shd''. - Read the ''msghdr_t'' header; iterate the trailing ''hfield_t'' records. - For each ''dfield_t'' entry of type ''TEXT_BODY'' (etc.), seek into ''.sdt'' and read the data, including its ''xlat[]'' prefix. - Apply translations in **reverse** order to recover the original data. - Concatenate body fields (if multiple). ===== SMBLIB ===== **SMBLIB** is the C library Synchronet uses to read and write SMB bases. Third-party programs may link against it (LGPL) rather than re-implement the format from spec. The library lives in [[https://gitlab.synchro.net/main/sbbs/-/tree/master/src/smblib|src/smblib/]] and exposes ~140 public functions covering opening/closing bases, allocating / freeing blocks, adding messages, reading messages, dump/utility helpers, and CRC/hash maintenance. Key files (see GitLab for current source): ^ File ^ Purpose ^ | [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbdefs.h|smbdefs.h]] | All format constants — field type codes, attribute flags, agent/network/media types, error codes, structure definitions. **The canonical reference for current values.** | | [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smblib.h|smblib.h]] | Public function prototypes | | [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smblib.c|smblib.c]] | Library implementation (open/close, headers, indexing) | | [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smballoc.c|smballoc.c]] | Block allocation (Self-pack, Fast, Hyper) | | [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbadd.c|smbadd.c]] | Adding messages | | [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbfile.c|smbfile.c]] | File-base entries | | [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbhash.c|smbhash.c]] | Hash file maintenance | | [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbstr.c|smbstr.c]] | String / header field helpers | | [[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/smblib/smbdump.c|smbdump.c]] | Dump utility helpers (used by ''[[util:smbutil]]'') | For working examples of message storage and retrieval, see ''[[util:smbutil]]'' (''[[https://gitlab.synchro.net/main/sbbs/-/blob/master/src/sbbs3/smbutil.c|src/sbbs3/smbutil.c]]'') and the relevant message-handling code in ''src/sbbs3/'' (''postmsg.cpp'', ''readmsgs.cpp'', ''writemsg.cpp''). ===== Performance Notes ===== * **Self-packing** is the default for typical sub-boards. It minimizes disk usage; import speed is bounded by allocation searches. * **Fast Allocation** trades disk usage for speed by appending; needs ''[[util:smbutil]] p'' periodically. * **Hyper Allocation** is fastest for large echomail imports but never reuses freed space — packing is mandatory. * **Mail base** must use Self-packing or Fast Allocation (Hyper is not supported). * **Duplicate checking** (''max_crcs'' > 0 or ''.hash'' enabled) adds a small per-message cost; disable on bases that don't need it (e.g. transient outbound bases). * **CRC index** keys (''to'', ''from'', ''subj'') let readers find messages by name/subject without opening every header — keep ''.sid'' in sync after operations. ===== Bibliography ===== * Original SMB v1.21 spec: [[https://synchro.net/docs/smb.html]] * SMBLIB source: [[https://gitlab.synchro.net/main/sbbs/-/tree/master/src/smblib]] * FidoNet Technical Standards: [[http://ftsc.org/]] * RFC 5322 (Internet Message Format): [[https://datatracker.ietf.org/doc/html/rfc5322]] * RFC 2045 / 2046 (MIME): [[https://datatracker.ietf.org/doc/html/rfc2045]] ===== Implementations ===== * Synchronet (BBS, mail server, NNTP server, gateways) — full implementation, all features * [[util:smbutil]], [[util:chksmb]], [[util:fixsmb]] — bundled utilities * Synchronet's [[server:mail|SMTP/POP3/IMAP server]] — SMB-backed * Synchronet's [[service:nntp|NNTP server]] — SMB-backed * Third-party readers and gateways may use SMBLIB or implement the spec independently ===== See Also ===== * [[ref:|Reference Library]] * [[util:smbutil]] — message base maintenance utility * [[util:chksmb]] — integrity checker * [[util:fixsmb]] — index rebuilder * [[ref:fidonet]] — FidoNet specifications * [[ref:qwk]] — QWK packet format * [[https://gitlab.synchro.net/main/sbbs/-/tree/master/src/smblib|SMBLIB source on GitLab]] {{tag>smb messagebase reference spec}}