Multi-delimiter tokenizer for SWMM input files. More...

#include <string>
#include <string_view>
#include <vector>

Include dependency graph for Tokenizer.hpp:

This graph shows which files directly or indirectly include this file:

Classes
class	openswmm::input::Tokenizer
	Stateless multi-delimiter tokenizer for SWMM input lines. More...

Namespaces
namespace	openswmm

namespace	openswmm::input

Detailed Description

Multi-delimiter tokenizer for SWMM input files.

Replaces the legacy space-only tokenizer in src/solver/input.c. Whitespace always delimits; a comma delimits only where it cannot be part of a name. This allows input files to be formatted as:

Traditional SWMM (space-delimited)
Spreadsheet export (comma or tab-delimited CSV)
Mixed formats within the same file

The comma rule is positional, not dialect-wide, because the two readings genuinely collide. Legacy splits on " \t\n\r" only, so a comma is an ordinary name character — and real decks depend on it: S-2-M20-L109/link2,6-201-9(u) is ONE subcatchment outlet. Treating every comma as a delimiter truncated that reference, read the name fragment after it as the subcatchment's area, and produced a trailing token the legacy reader then rejected as an undefined object.

So: a comma separates when it sits at a token boundary — adjacent to whitespace or at a line edge — or when the line carries no whitespace at all outside quotes and is therefore pure CSV. A comma with non-whitespace on both sides, in an otherwise whitespace-delimited line, belongs to its token. A value that genuinely contains a comma AND sits in a CSV row must be quoted, which is the ordinary CSV convention.

Key differences from legacy SWMM tokenizer

Feature	Legacy (input.c)	New Tokenizer
Delimiters	Space, tab, CR/LF	Same, plus comma at a token boundary
Comment char	`;` (semicolon)	`;` or `;;`
Quoted strings	Filenames only	Whole tokens or embedded key="value" (double-quotes)
Performance	sscanf-based	Reusable view vector avoids per-token copies

See also: Legacy reference: src/solver/input.c — getToken(); tests/unit/test_tokenizer.cpp

Author: Caleb Buahin caleb.nosp@m..bua.nosp@m.hin@g.nosp@m.mail.nosp@m..com

License: Apache-2.0

Classes

Namespaces

Detailed Description

Key differences from legacy SWMM tokenizer