# TriasDev.Tabular > Fast, low-memory reading of large Excel, OpenDocument and CSV files for .NET — profile every column, map, validate and import, with no dependencies TriasDev.Tabular is a .NET 8/10 library (NuGet: TriasDev.Tabular, no dependencies) that reads csv, xlsx, ods and zip archives of them, profiles every column over every row, and imports rows through a column mapping a person confirmed. The flow is two independent reads of the same file, with no state kept between them: 1. `TabularFile.Open(stream, name)` returns an `ITabularCursor`; `new TabularAnalyzer().Analyze(cursor)` returns a `FileProfile` (per sheet `SheetProfile`, per column measured `ColumnFacts` and ranked `TypeHypothesis` suggestions). 2. Declare fields with `ImportField.Text/Integer/Decimal/Date/Boolean(...)` and rules (`Require()`, `MaxLength()`, ...); put them in a `TargetSchema`; build a `MappingPlan` (`MappingPlan.ByHeader(sheetProfile, schema)` or by hand); optionally judge it with `MappingPrecheck.Check(plan, schema, profile)`. 3. `TabularImporter.Import(stream, name, plan, schema, row => new MyType(row[field], ...))` returns an `ImportRun`: iterate outcomes (value or `RowError`s), or use `InChunks` / `All(limit)`. Rules that are easy to get wrong: a row (`CurrentRow`, `ImportRow`) is a view over a reused buffer — copy what you keep before the next read. Row errors carry codes such as `value.required`, not messages. Library exceptions derive from `TabularException` and carry a code (`TabularFormatException`, `TabularLimitException`, `TabularStructureException`, `MappingPlanException`). Every structure read from a file is bounded by an option. Streams are closed by the library unless `LeaveOpen` is set. The format is decided by the file's bytes, never its name. # Start # TriasDev.Tabular **Fast, low-memory reading of large Excel, OpenDocument and CSV files for .NET — with no dependencies.** Hand it a file and it tells you what is in it: every column's type, emptiness, uniqueness, value ranges and the rows that do not fit — measured over every row, not a sample. Then read the data itself, as raw cells or as typed rows through a mapping, with errors that point at the row and column they came from. It reads multi-million-row files in seconds while its memory stays flat as the files grow, and it is built on the base class library alone: no third-party packages. ``` dotnet add package TriasDev.Tabular ``` ## Choose your path ### I am adding an upload to my application A user uploads a spreadsheet; you profile it, let them map its columns to your fields, check the mapping, and import typed rows or precise errors. - [Getting started](https://triasdev.github.io/tabular/getting-started/index.md) — install, profile, map, check, import - [How it works](https://triasdev.github.io/tabular/concepts/index.md) — two independent reads of the file, facts against suggestions - [Importing](https://triasdev.github.io/tabular/importing/index.md) — the run, batches, rules, translated fields, the precheck - [Error codes](https://triasdev.github.io/tabular/error-codes/index.md) — every code a row error or an exception carries ### I just need to read big files fast A forward-only cursor over rows of typed cells, for xlsx, ods, csv and zip archives of them. - [Formats](https://triasdev.github.io/tabular/formats/index.md) — what each kind of file reads as, and how malformed csv is repaired - [Performance](https://triasdev.github.io/tabular/performance/index.md) — the library's own numbers - [Benchmarks](https://triasdev.github.io/tabular/benchmarks/index.md) — against Sylvan, Sep, CsvHelper, ExcelDataReader, MiniExcel, ClosedXML, NPOI and EPPlus ### I need to know what it will refuse - [Bounds](https://triasdev.github.io/tabular/bounds/index.md) — every ceiling that protects a server from a hostile file - [Streams, cancellation, progress and cultures](https://triasdev.github.io/tabular/operations/index.md) - [Known issues](https://triasdev.github.io/tabular/KNOWN-ISSUES/index.md) — behaviour at the edges, with what would make each one matter ## For AI agents The documentation is also published for language models, following [llms.txt](https://llmstxt.org): [`llms.txt`](https://triasdev.github.io/tabular/llms.txt) is a short map of the library, and [`llms-full.txt`](https://triasdev.github.io/tabular/llms-full.txt) is every page in one Markdown file. Each page is also available as Markdown next to its HTML. ## Source [github.com/TriasDev/tabular](https://github.com/TriasDev/tabular) · MIT licence · [NuGet](https://www.nuget.org/packages/TriasDev.Tabular/) # Getting started ## Install ``` dotnet add package TriasDev.Tabular ``` The package targets .NET 8 and .NET 10 and references nothing but the base class library. ## Read a file `TabularFile.Open` decides the format from the file's bytes — csv, xlsx, ods, or a zip archive of them — and returns a cursor over its sheets and rows: ``` using TriasDev.Tabular; using FileStream file = File.OpenRead("orders.xlsx"); using ITabularCursor cursor = TabularFile.Open(file, "orders.xlsx"); foreach (SheetInfo sheet in cursor.Sheets) { cursor.MoveToSheet(sheet.Index); while (cursor.ReadRow()) { ReadOnlySpan row = cursor.CurrentRow; // typed cells: text, number, date, boolean } } ``` A row is a view over a reused buffer: read what you need from it before the next `ReadRow`. ## Import into your own type The steps an upload goes through, from [`samples/TriasDev.Tabular.Samples.Import`](https://github.com/TriasDev/tabular/tree/main/samples/TriasDev.Tabular.Samples.Import), which the build compiles — so the code on this page is code that works. **Declare the fields once.** They are both the schema and the accessors that read a row: ``` // The fields, declared once: they build the schema and read the values. TextField name = ImportField.Text("name").Require().MaxLength(100); TextField country = ImportField.Text("country").ExactLength(2); DateField signedOn = ImportField.Date("signed_on"); DecimalField amount = ImportField.Decimal("amount").Require(); TargetSchema schema = new() { Fields = [name, country, signedOn, amount] }; ``` **Profile the file and build a plan from its headers.** In an application, a person confirms or changes the plan on a mapping screen; the profile is what that screen shows: ``` // 1. Profile the file, and build the plan from its headers. FileProfile profile; using (FileStream file = File.OpenRead(path)) using (ITabularCursor cursor = TabularFile.Open(file, Path.GetFileName(path))) { profile = new TabularAnalyzer().Analyze(cursor); } MappingPlan plan = MappingPlan.ByHeader(profile.Sheets[0], schema, culture: "de-DE"); ``` **Check the plan against the profile**, before the file is read again: ``` // 2. Judge the plan against the profile, before reading the file again. PrecheckResult check = MappingPrecheck.Check(plan, schema, profile); foreach (PrecheckFinding finding in check.Findings) { Console.WriteLine($"precheck: {finding.Severity} {finding.Code} on {finding.TargetFieldName}"); } if (!check.CanImport) { return 1; } ``` **Import:** typed rows, or errors that name their row, column and code: ``` // 3. Import: typed rows, or errors that point at their row and column. using FileStream again = File.OpenRead(path); using ImportRun run = TabularImporter.Import( again, Path.GetFileName(path), plan, schema, row => new Customer(row[name]!, row[country], row[signedOn], row[amount]!.Value)); foreach (ImportOutcome outcome in run) { if (outcome.HasErrors) { foreach (RowError error in outcome.Errors) { Console.WriteLine($"row {error.RowNumber}, {error.TargetFieldName}: {error.Code} ({error.RawValue})"); } continue; } Customer customer = outcome.Value; Console.WriteLine(string.Create( System.Globalization.CultureInfo.InvariantCulture, $"imported {customer.Name}, {customer.Country}, {customer.SignedOn:yyyy-MM-dd}, {customer.Amount}")); } Console.WriteLine($"{run.Summary.RowsProduced} imported, {run.Summary.RowsFailed} failed"); ``` Run it with `dotnet run --project samples/TriasDev.Tabular.Samples.Import`: the sample file has a date that is not one and an empty required amount, and both come back as row errors. ## Next - [How it works](https://triasdev.github.io/tabular/concepts/index.md) — why analysis and import are two reads, and what a profile measures - [Importing](https://triasdev.github.io/tabular/importing/index.md) — batches, rules of your own, translated fields, the precheck in depth - [Formats](https://triasdev.github.io/tabular/formats/index.md) — what csv, xlsx, ods and zip files read as # How it works Reads an Excel, OpenDocument or CSV file — or a zip of them — reports what is in it, and extracts it through a mapping a person confirmed. The library **declares no package reference of its own**: everything it does with a file, it does with the base class library. The build adds analyzers, and nothing else; none of them is referenced by any code here. `TabularIndependenceTests` checks the resolved package graph, not just the project file, so a parsing library arriving through a transitive reference fails a test. ``` Analyze file ─────────────────────────► FileProfile sheets, columns, measured facts, ranked readings, detected dialect a person maps columns to fields, in a UI this library knows nothing about Extract file + MappingPlan + schema ──► typed rows, or errors that locate themselves ``` The two are **independent reads of the file**. The library keeps no state between them and has no persistence; a caller that wants to hold a profile while a user thinks stores it itself. ## Analysis reads every row Not a sample. The questions a mapping screen has to answer are falsified by a single row anywhere: > This column holds ISO-3166 country codes. It must be exactly three characters, never a number, and may be empty. One four-character value in the forty-thousandth row *is* the answer, and no sample finds it. ## Facts and hypotheses are different things - **`ColumnFacts`** is measured. Counts, lengths, ranges, how values fare under each culture, how many are distinct, where the ones that did not parse stand. - **`TypeHypothesis`** is derived, ranked, and carries a confidence and the located outliers. It is a suggestion for a UI — it pre-selects a mapping and shows the risk. It never decides anything. Why that matters, from a real file: a column of `1,00 2,00 3,00` yields a decimal hypothesis at full confidence. That is arithmetically perfect and practically useless — the column is an identifier, and only the facts beside it (every value distinct) say so. ## What belongs to the sheet Each `SheetProfile` says what its own source was: `Format`, `Source` (a path inside an archive, else null), the csv `Dialect` it was read with, and the `Diagnostics` of what was repaired in it. `FileProfile.Format` is the container's; `FileProfile.Diagnostics` is every sheet together. Both are snapshots taken when the pass ended. # What it deliberately does not do - **Guess where the header is.** The first row is the header. A guess that is usually right produces a wrong answer nobody checks; `MappingPlan.HeaderRowIndex` is where a user says otherwise. - **Decide a column's type.** It proposes. A person disposes. - **Distinguish an empty field from a quoted empty one.** The syntax does; the data does not. - **Scan for viruses.** That belongs before a file reaches here. - **Offer a transformation language.** A list of empty-equivalents, and nothing more, because nothing yet asks for more. - **Count distinct values beyond its budget.** Past it, a column reports a lower bound and an undetermined uniqueness — an explicit "not determined" rather than a confident wrong number. # Using it # Importing ``` using FileStream file = File.OpenRead(path); using CsvCursor cursor = new(file, Path.GetFileName(path)); FileProfile profile = new TabularAnalyzer().Analyze(cursor); // hand `profile` to a mapping screen, or build the plan from the headers (below) ``` ``` // Declared once. These same objects build the schema and read the values, so a field's name is // written in exactly one place. static class Fields { public static readonly TextField Name = ImportField.Text("name").Require().MaxLength(255); public static readonly TextField Country = ImportField.Text("countryCode").Require().ExactLength(2); public static readonly DateField Signed = ImportField.Date("signedOn"); } static readonly TargetSchema Schema = new() { Fields = [Fields.Name, Fields.Country, Fields.Signed] }; // One expression turns a row into your own type. It knows nothing about cursors, positions, or the // order the schema declares. static Customer Build(ImportRow row) => new() { Name = row[Fields.Name], Country = row[Fields.Country], SignedOn = row[Fields.Signed], }; ``` The plan says which column feeds which field. A mapping screen builds it from the profile; where the columns are named after the fields, `MappingPlan.ByHeader` builds it without one — headers compared ignoring case, spaces and `_ - .`, or by a rule of your own (a synonym list, say). Unmatched fields stay unbound, for `MappingPlanValidator` to report if they are required. ``` MappingPlan plan = MappingPlan.ByHeader(profile.Sheets[0], Schema, culture: "de-DE"); ``` A plan built by `ByHeader` also records the sheet's name and source. Extraction checks them against the sheet at the plan's index and refuses another one (`structure.sheet-changed`), so a workbook whose tabs were reordered is not imported from the wrong tab; the precheck blocks such a plan with the same code. A plain csv's name is not recorded — it is whatever name the caller passed in, and the file has one sheet anyway. A plan written by hand can leave both null. ``` using FileStream file = File.OpenRead(path); using ImportRun run = TabularImporter.Import(file, Path.GetFileName(path), plan, Schema, Build); foreach (ImportOutcome outcome in run) { if (outcome.HasErrors) { Report(outcome.Errors); continue; } Persist(outcome.Value); } ExtractionSummary summary = run.Summary; ``` Importing a different kind of entity is a different `Fields`, a different `Schema` and a different `Build`. Nothing else changes. ## What a run reports about itself ``` run.Summary // rows read, produced, skipped, failed; whether it stopped early run.Coverage // per field: how many rows carried a value, and the share run.Preview // the first N rows as mapped, when PreviewRows asked for any ``` `Coverage` is counted for every run, because it costs one check per field and answers the question an error list cannot: whether the columns a person mapped actually carry anything. A field bound to the wrong column is not *invalid* — it is empty, so it passes every rule and shows up nowhere else. `Preview` is off by default. A screen showing a person what an import will produce asks for a handful; a nightly load of five million rows has nobody to show them to. ## Three ways to read a run, over one core ``` foreach (ImportOutcome outcome in run) { } // a row at a time foreach (ImportChunk chunk in run.InChunks(100_000)) { await repository.BulkInsertAsync(chunk.Items); // a batch at a time Report(chunk.Errors); } ImportResult result = run.All(limit: 50_000); // all of it, with a ceiling ``` Five million rows are not five million inserts, which is what `InChunks` is for: each batch carries what was built and the failures from the same window, so the report keeps pace with the writing. `All` holds a ceiling because a convenience that quietly consumes a machine is not a convenience. ## If you need the values rather than an entity `TabularExtractor.Start` is the layer underneath, and hands back typed values without building anything. `TabularImporter` is that plus your mapper, and is what a caller normally wants. A workbook, an OpenDocument spreadsheet or a zip archive is the same call. Which kind of file it is comes from its bytes, not its name — a csv saved as `.xlsx` is commoner than it ought to be, and a reader that trusts the extension fails on it with a message about a corrupt archive. A zip's directory says whether it is a workbook, a spreadsheet or an archive of files. ## Checking a mapping before importing through it The file was profiled once, over every row. Asking what that already measured is cheaper than learning it again row by row, after some records have been written: ``` PrecheckResult check = MappingPrecheck.Check(plan, Schema, profile); if (!check.CanImport) { return Refuse(check.Findings); } ``` It answers what a row cannot. Whether a column's values are all different is a property of the column, so no per-row rule can decide it — `ImportField.Text("cid").Require().Unique()` is settled here, exactly, because the distinct values were already counted. A finding says *these rows will fail*, or *no row can succeed*, or *this cannot be judged from here*. It never says the rest is fine: an allowed-value set is measured against a bounded sample, and a rule spanning two fields is invisible to facts about one. `TargetSchema.Policy` decides what a partial success means. `BestEffort` imports what fits; `AllOrNothing` refuses a file that would import partially. That is the programmer's call, not the uploader's — whether half an import beats none depends on what is being imported, and only whoever declared the target knows. Under `AllOrNothing` the precheck blocks on any finding it is sure of, and the run itself stops at the first row that fails (`Summary.StoppedEarly`). `All()` then returns no items, only the error, and the batch from `InChunks` that holds the failure carries no items. What a streaming run cannot do is take back batches it handed out before the failure: a caller writing batch by batch commits once, at the end, or rolls back. ## Two rules a caller must know **Rows are views, not copies.** `ImportRow`, `CurrentRow` and `CurrentValues` look at memory that is overwritten on the next read. `ImportRow` is a `ref struct`, so the compiler will not let one escape a mapper; the other two are yours to copy from if you keep anything. Handing out a fresh array per row would cost an allocation per row of a file that may hold millions. **A field belongs to a schema.** Asking a row for a field the schema does not declare throws, on the first row. A field left over from another target, or renamed in one place and not the other, is a defect in the caller and reads as one instead of arriving as an empty column. **A row is either values or errors, never both.** Half a row invites half an entity, which is how silent corruption starts. ## A field the file says in several languages A catalogue carries `Title#en` beside `Title#de` — two columns saying one thing. Declared once: ``` public static readonly TranslatedField Title = ImportField.Translated("title", ["en", "de"]).Require(); ``` Underneath these are ordinary fields with ordinary names, `title.en` and `title.de`, each fed by one column. That is the point of the shape: a binding, a plan and a row are exactly what they were, so nothing downstream learns about languages. What the group adds is the three things the two columns have in common — a screen can draw them together, `Required` means *at least one of them* rather than each, and the mapper gets them back as one value: ``` TitleTextValues = row.Translations(Fields.Title) // { "en": "…", "de": "…" } ``` A language the row left empty is absent from that dictionary rather than present and empty: "not translated" and "translated to nothing" are different things to whatever stores the result. `Required` on a group is deliberately weak. A catalogue translated into German alone is a complete catalogue, and a rule naming English would refuse it for saying nothing wrong. A row carrying none of the languages fails once, as `group.required`, rather than once per declared language. The precheck reaches one of the two conclusions here and says so. If every column mapped to the group is empty from top to bottom, no row can carry any of them — that is certain, and it blocks. Whether *some particular row* leaves all of them empty is not: two columns can be empty in complementary halves of a file and cover every row between them. That is settled per row, while importing. Where the file marks its languages in the headers, `HeaderVariant.TryParse` reads the mark — `#`, `_`, `-`, `.`, `@`, a space, or brackets — but only for a variant the group declares, so `Order_id` stays one field. It proposes; it never decides. A header is written by whoever exported the file, and acting on it silently is the worst mistake available here: the text arrives, it is simply filed under the wrong language, and nobody finds out until a reader of that language does. ## Rows the run drops A row blank from end to end is padding — a spreadsheet accumulates it below the data as a matter of course — and it is skipped without comment. A row that carries a value only in a column nobody mapped is a different thing: a record the file contains and the run drops. `Summary.RowsWithNothingMapped` counts those separately. Counted rather than failed. Faulting them would bury a real import under errors about footnotes, and skipping them silently is the worse mistake: data disappearing quietly is harder to notice than a number that does not add up. ## What the precheck can and cannot settle It answers from the profile, which was measured before anybody built a mapping. So it can only be right about the import if it measured what the import will do, and three things used to make that false — every one of them producing findings about values no row would ever hold. Trimming is part of reading now, unconditionally, so `" DE "` is two characters to both halves of the library rather than four to one of them. It was a per-binding option applied only while extracting, which is a disagreement waiting to happen rather than a feature: leading whitespace in a cell is an artefact of how it was typed, not a value. A binding's empty-equivalents are subtracted before a column is judged, because `k.A.` is not a country that failed to be allowed — it is the file saying it has nothing to say. Where they are declared, the count of rows leaving a required field empty becomes *at least* N, since those rows were measured as values and will be read as absent. And a profile carries the header row it was measured against. When a mapping names a different one, every fact describes a different file — the real header and everything above it were counted as data — so the precheck reports `mapping.stale-profile` and judges nothing. Undetermined rather than blocking: the file is very likely fine and it is the profile that is stale. Analysing it again under the chosen row is what settles it. Every rule is judged the same way: each of the column's distinct values is put back into the cell it came out of and read exactly as the extractor would read it — same reader, same cell, same culture, same type — and then the rule itself is asked. Four separate judges stood here before, three of which compared the file's characters while the import compared what it renders from them. A pattern of three digits passed `007` and then failed every row, because the import sees `7`. *Same cell* is not decoration. A workbook types its own cells and the extractor takes such a cell at its word; the profile keeps distinct values as text, so re-reading that text under the mapping's culture answers a question the import never asks. A native `1234.5` read back under German — where the point is a group separator — becomes 12345, and the precheck refused a file that imports perfectly. That is the same defect as the one above, closed for csv and reopened for workbooks, and it is why the value is rebuilt into its declared kind first. That is affordable because it runs once per *distinct* value rather than once per row, and it is only attempted where the profile kept every distinct value. Where it did not, nothing is claimed. The header a binding recorded is compared with the one the column now carries, because extraction refuses the whole run over that and the precheck used to be the only thing that could not see it. One more rule governs when a finding may say *no row can succeed*: only when every row carries a value. Facts about values are measured over the non-empty cells and a rule about values is never applied to a cell without one, so a column of ten thousand blanks and one bad value fails one row and imports the rest — unless the field is required, which makes an empty cell a failure on its own account. ## Asking whether a column holds your reference data Length and type say a column *could* be a country code. They cannot say it *is* one — `ZZ` is two characters and parses as text exactly like `DE`. That question is about the values, and it is answered against a set the library never sees: ``` ImportField.Text("countryCode").ExactLength(2).AllowedValues(CountryCodes) ``` Declared once, it is enforced twice, and the two are not the same thing: - **On import**, per row: a row carrying `ZZ` is refused with `value.not-allowed`. That is the guarantee. - **In a precheck**, per *distinct value*: "2 of 5 distinct values are not allowed: QQ, ZZ". That is the suggestion, and it is what lets a screen say *this is your country column* — or *this is not*, when every value is a stranger. The second is affordable because a column of codes is low-cardinality by nature — ISO 3166 has some 250 members, the ELF list of legal forms some 2,600 — so the work is bounded by the column's variety rather than by the file's length. `ColumnFacts.DistinctValues` carries them, and `DistinctValuesAreComplete` says whether it is the whole set. When it is not, the precheck answers `Undetermined` rather than drawing a conclusion from a subset: a column with fifty thousand distinct values is not a column of codes, and the per-row check is what stands behind it. Measured cost of keeping them, against not keeping them — same binary, one option apart, best of seven for the small files and of three for the large one: | file | rows | time, off → on | allocated, off → on | | ---------------- | --------- | --------------- | -------------------------------- | | 10k-row workbook | 10,007 | 269 → 273 ms | 18.52 → 18.66 MB (+0.8%) | | 10k-row csv | 10,000 | 168 → 170 ms | 12.10 → 12.25 MB (+1.2%) | | 5M-row csv | 5,127,959 | 22.60 → 22.42 s | 8,372.50 → 8,372.70 MB (+0.002%) | The time differences are inside run-to-run variance in both directions. The memory cost does not grow with the file — it is a few hundred kilobytes for a thousand strings per column, spent once — which is the property that makes the check affordable at all: it falls on how varied a column is, never on how long the file is. ## Rules of your own A pattern checks an identifier's shape; many identifiers also carry a check digit that a pattern cannot see. `Must` declares a rule from a predicate, under a code of the caller's choosing: ``` public static readonly TextField Isin = ImportField.Text("isin") .Require() .ExactLength(12) .Must("isin.check-digit", CheckDigits.Luhn); public static readonly TextField Lei = ImportField.Text("lei") .ExactLength(20) .Must("lei.check-digits", CheckDigits.Mod97); ``` It is applied wherever the built-in rules are: to every row at import, where a failure is a `RowError` carrying that code and the cell's location, and in `MappingPrecheck` to the column's distinct values — "2 of 1,200 distinct values fail this rule" — or `Undetermined` where the profile did not keep them all. It is only asked about a value that is present, and typed by its field: `Func` for text, `long` for integers, `decimal`, `DateTime`. The code is yours to translate and may not start with `value.`, `mapping.`, `group.` or `structure.`, so a caller's rule is never mistaken for one of the library's codes below. `CheckDigits` ships Luhn (card numbers; ISINs, with letters counted as A = 10 … Z = 35) and ISO 7064 MOD 97-10 (LEIs; IBANs with their first four characters moved to the end). # Formats Which kind of file it is comes from its bytes, never its name. What each kind reads as: ## An OpenDocument cell says its own type A workbook guesses dates from number formats; a `.ods` cell states its type beside its value, so there is nothing to guess. `float`, `percentage` and `currency` read as numbers, `date` as a date, `time` as the workbook serial of as many days — a time of day on 31 December 1899, the day an xlsx time-only cell reads on, and a longer duration on the day that serial names — and `boolean` as a boolean. Anything else is text: the cell's `office:string-value` when it has one, else its paragraphs joined by a line feed, comments left out. A formula reads as the value the writer cached. ODF has no error type; LibreOffice marks a failed formula in an extension attribute, and it reads as an error carrying the text the cell shows — `#N/A`, `#REF!`, or LibreOffice's own `Err:502`. A row or cell repeated by attribute is expanded only when it holds a value; the million empty rows LibreOffice declares after the last one cost nothing. Covered cells of a merge read as empty. Hidden sheets and rows read like any other, as in xlsx. A sheet is a table of the spreadsheet itself: a sub-table inside a cell, or the table a DDE link caches, is not one, and its text is not the cell's. ## A zip archive reads as one workbook A zip that is not itself a workbook is read as one: its sheets are the sheets of every file in it that can be read as a table — csv, xlsx and ods — in the order of their paths, each with its file's path as `SheetInfo.Source` and its own `Format`. A csv file is named after its file; a workbook's sheets keep their names, and `Source` tells two `Sheet1` apart. Files are judged by their bytes, as a file on its own is. - **Left out without a word:** directories, hidden files and folders (`.DS_Store`, anything whose name starts with a dot) and `__MACOSX/`. - **Skipped with a reason** in `FileProfile.SkippedEntries`: an encrypted file, a nested zip, another OpenDocument type, a legacy `.xls`, an XML document, a binary file, and a workbook that is damaged or of a kind not read (`.xlsb`). Nested archives are not opened. - **Refused:** an archive with nothing readable in it, as `format.unsupported`. A csv file is read as a stream straight out of the archive, never unpacked, with its dialect decided from its own head; moving back to it reads it again from its first row. A workbook has to be read with random access, so it is copied into memory while its sheets are read — one at a time, up to `ArchiveCursorOptions.MaxEmbeddedWorkbookBytes` (256 MB by default, and any size a server can afford: the copy is held in pieces, not one array). A mapping plan made from an archive records the sheet's `Source`, and an import refuses the archive as `structure.sheet-changed` when another file now stands at the plan's index. ## Malformed input is repaired, and the repair is counted Files that people upload are not well-formed. A 572 MB real-world export carries quotes inside unquoted fields (`100 21"st AVE`), text after a closing quote (`"C" Road`), and 302 quotes that are never closed. Refusing such a file is defensible and useless; one bad address must not fail an import of five million rows. So the reader recovers — and counts what it recovered in `CursorDiagnostics`. A silent recovery is indistinguishable from correct reading, and *that* is the defect: left unbounded, those 302 quotes swallow about 39,000 records without a word. Two repairs, both counted. A quoted field that has crossed a line ending and holds a whole record's worth of delimiters was never a quote — a lone `"` that opened a field and swallowed the records after it — and those records are read as records the moment that is clear (`RecoveredStrayQuotes`). Past a line ending a quote also closes a field only where a field can end, so an inch mark in the swallowed text (`135"th`) does not close it. Behind that, a quoted field longer than 100 lines, or one the file ends inside, is abandoned the same way (`RecoveredUnterminatedQuotes`); that bound is what protects tables narrower than five columns, where the delimiter test is off. Genuine multi-line values — an address, a long note — hold a delimiter or two at most and are left alone. # Streams, cancellation, progress and cultures ## Who closes the stream One rule at every entry point: a stream handed over is closed — when the cursor or run is disposed, and also when the call fails — unless the caller asked for it to stay open. | Entry point | Closes the stream | Keep it open with | | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | ------------------------------ | | `new CsvCursor(stream, …)`, `new XlsxCursor(stream, …)` | on `Dispose`, or on a failed open | `leaveOpen: true` | | `TabularFile.Open(stream, …)` | on `Dispose`, or on a failed open | `TabularOpenOptions.LeaveOpen` | | `TabularImporter.Import(stream, …)` | on `Dispose` of the run, or when the call throws (a refused plan included) | `ImportOptions.Open.LeaveOpen` | | `TabularImporter.Import(cursor, …)`, `TabularAnalyzer.Analyze`, `TabularExtractor.Start` | never — the cursor is the caller's | — | `TabularOpenOptions` also carries the csv, xlsx and ods cursor options and the archive's bounds, so ceilings can be changed without giving up format detection. The files inside an archive are read with the same csv, xlsx and ods options as files on their own. ## Cancellation `ReadRow` takes a `CancellationToken`, and the analyzer and importer hand theirs down rather than only checking between rows. That distinction is the whole of it: the expensive things happen *inside* a single read — a shared string table of a million entries loads on the first cell that refers to one — and a check between rows never runs while that is happening. A timeout at the request level cannot reach a thread that is inside such a call. The token is checked on a stride rather than per character, because the check is cheap but not free and a row is normally over in a few hundred characters. Every operation that reads takes a token, last parameter, as the BCL's do: `Open`, `Analyze`, `Start` and `Import` for the work they do up front, and `ExtractionSession.ReadRow`, `ImportRun.Rows`, `InChunks` and `All` for the reading. The token a run was started with keeps applying to every read of it, so either one stops the run — a plain `foreach` over the run, which cannot pass a token, is stopped by the run's. ## Progress Analysis of a multi-million-row file takes seconds to tens of seconds, and a screen waiting on it wants to say how far it has got: ``` IProgress progress = new Progress(p => Console.WriteLine($"{p.SheetName}: {p.RowsRead:N0} rows, {p.Fraction:P0}")); FileProfile profile = new TabularAnalyzer().Analyze(cursor, progress, cancellationToken); ``` The fraction is taken from how much of the file the reader has consumed — a csv's stream position against its length, a workbook's worksheet bytes against their total, which the package directory states up front — so nothing reads the file twice to have a denominator. It is exactly 1 in the final report, which has `IsComplete` set; where the stream has no length it is null until then. Reports go out when the fraction has moved by `AnalysisOptions.ProgressStep` (1% by default) **and** at least `ProgressInterval` data rows (10,000) have passed since the last one. A five-million-row file reports about a hundred times; a file of twenty thousand rows, read in milliseconds, once or twice. With no length to measure, the row interval alone decides. `Progress` posts each report to the context it was created on; an `IProgress` of your own is called on the analysing thread. A console progress bar with Ctrl+C cancellation, runnable: [`samples/TriasDev.Tabular.Samples.Progress`](https://github.com/TriasDev/tabular/tree/main/samples/TriasDev.Tabular.Samples.Progress). ## Cultures Every text value is tried under each culture in `AnalysisOptions.Cultures`, by default `["", "de-DE", "en-US"]` — invariant, German and US conventions — and the ranked hypotheses name the culture that read a column — the empty string for the invariant one, the same spelling `MappingPlan.Culture` takes, so a hypothesis's culture goes into a plan as it is. The default leans towards the files this library was first written for; a caller whose files come from elsewhere should list its own (`["", "fr-FR"]`). An unknown name is refused when the analyzer is created. Under invariant globalization (`InvariantGlobalization` / `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT`, common in slim container images) only the invariant culture exists. Analysis then leaves the named cultures out instead of failing — the profile's `ParseCounts` show which cultures were used — and a mapping that names one is refused by the validator and the precheck as `mapping.unknown-culture`. German amounts such as `1.234,50` cannot be read as numbers in that mode. # Reference # Error codes The library reports codes and never messages: it knows nothing about who reads them or in what language. A calling domain maps them onto its own error envelope, and a frontend derives its wording from them. | Code | Meaning | | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `value.required` | A required field's cell was empty | | `value.type-mismatch` | The value does not read as the field's type | | `value.exact-length`, `value.min-length`, `value.max-length` | A length rule | | `value.out-of-range` | A minimum or maximum | | `value.not-allowed` | Outside the allowed set | | `value.not-unique` | The column repeats a value, and the field identifies a record | | `value.pattern` | Did not match the pattern (as a whole: patterns are anchored at both ends) | | `group.required` | A row carries none of a group's variants | | `mapping.unknown-field`, `mapping.duplicate-binding`, `mapping.required-field-unmapped`, `mapping.required-group-unmapped` | A plan that does not fit its schema | | `mapping.invalid-column`, `mapping.invalid-header-row`, `mapping.invalid-sheet`, `mapping.unknown-culture` | A plan that is malformed | | `mapping.constraint-type-mismatch` | The schema puts a range (`MinValue`/`MaxValue`) on a field that is not a number | | `mapping.stale-profile` | The profile was measured against a different header row | | `mapping.header-changed` | The column's header is not the one the mapping recorded | | `mapping.invalid-plan` | `MappingPlanException`: the plan does not fit its schema; its `Faults` carry the codes above | | `structure.sheet-missing`, `structure.sheet-changed`, `structure.header-row-missing`, `structure.header-changed` | `TabularStructureException`: the file is not the one the plan was built for | | `format.unsupported`, `format.corrupt`, `format.truncated` | `TabularFormatException`: not a format this library reads (.xls, .xlsb, .fods, another OpenDocument type, binary, an archive with nothing readable), or damaged, or cut off | | `limit.exceeded` | `TabularLimitException`: a bound was exceeded; `Limit` names the option, `Maximum` its value | This table is checked against the library's sources by `ErrorCodeCatalogTests`, in both directions. It went out of step twice in the branch that added it — a code emitted, asserted, given a requirement and described in this file's own prose, and left out of the table a frontend reads. Now it cannot. Faults that invalidate a whole run are exceptions, not row errors — the two demand opposite responses. All of them derive from `TabularException`, which carries a `Code` from the table, and split by what a host does about them: | Exception | Means | Typical HTTP answer | | --------------------------- | ------------------------------------------------------------------------- | ------------------- | | `TabularFormatException` | Not a file this library reads, or not a readable one | 400 / 415 | | `TabularLimitException` | Readable, but beyond a configured bound — how most hostile files end | 413 | | `TabularStructureException` | Not the file the plan was built for (sheet, header row or header changed) | 409 / 422 | | `MappingPlanException` | The plan does not fit its schema, before any file is read | 400 | Mistakes in the calling code — a null argument, an option out of range, a field the schema does not declare — are `ArgumentException` and `InvalidOperationException`. Nothing else escapes: malformed XML and a damaged zip are reported as `TabularFormatException` with the parser's error as the inner exception. # Bounds | | Default | Why | | ---------------------------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Archive entries | 16,384 | The directory is read before anything else, and each entry costs a sniff | | Archive expansion | 8 GB | Every entry's declared size together; an archive may carry a zipped ods at that format's own budget | | Workbook inside an archive | 256 MB | Held in memory while its sheets are read, one at a time. A server may raise it to any size, past 2 GB too | | Package expansion | 2 GB (ods: 8 GB) | A zip's ratio is unbounded by design; a 50 MB upload could otherwise become fifty gigabytes. OpenDocument writes about four times the bytes for the same cells, so its budget is four times larger | | Package parts | 16,384 | Every entry's metadata is materialised to find parts by name, before any budget can be consulted | | Worksheets | 4,096 | One descriptor per sheet, held for the cursor's life and walked by anything that analyses the file | | Workbook relationships | 8,192 | A map built before anything reads from it; a workbook declares about one per sheet | | Shared string entries | 1,048,576 | The sheet's own row count — more distinct strings than a column can hold cells | | Shared string characters | 64 M | What a table costs is its characters, not its entries — a million entries of two thousand each is 3.9 GB | | Cell formats | 100,000 | Read in the constructor, before a caller has anything to cancel with; the format itself stops at 65,490 | | Cell value length | 16 M chars | A value is assembled from as many small runs as a file cares to write, none of them large | | Package metadata string | 2,048 chars | A sheet name, a relationship target, a format code — each had a ceiling on how many, none on how long | | Columns per row | 16,384 | The workbook format's own width. A csv has none, and a file of nothing but delimiters is the cheapest attack there is | | Cells repeats hand out (ods) | 100,000,000 | The row and column ceilings bound a sheet's shape, not the work: one cell repeated across every column of a row repeated a million times is under a kilobyte and seventeen billion cells. The copies count, across the file | | Rows holding a value (ods) | 1,048,576 | OpenDocument repeats a row or cell with one attribute. An empty repeat only moves the position along; one that holds a value is expanded, so it is bounded here and by the column ceiling — twenty characters of markup could otherwise ask for a billion cells | | Field length (csv) | 16 M chars | Held twice while a quoted field is open, once as the value and once as the text kept for a replay | | Quoted field length | 100 lines | Beyond that an opening quote was never syntax. In a table of five columns or more a stray quote is caught sooner, by swallowing a record's worth of delimiters | | Distinct tracking | 2,000,000 values | Exact counting costs memory in proportion; the budget is per file, not per column | | Retained distinct values | 1,000 per column | Enough to judge a column of codes against a reference set; a column with more is not one | | Error rows | 1,000 | A wrong mapping fails every row, and the thousand-and-first error says nothing the first did not | | Rows per sheet | 2,147,483,647 (csv), 1,048,576 (xlsx) | Row numbers and counts are `int`: the workbook format stops at a million rows, and a csv that long is some 100 GB | Getting these right took three attempts, and the pattern of the mistakes is worth more than the numbers. The package budget counts bytes a part expands to, which is not what those bytes become in the heap — so a shared string table needed a ceiling of its own, and the first such ceiling counted entries, which a million two-thousand-character entries satisfy at a cost of 3.9 GB. Then a review found three more collections growing from the file with no ceiling at all, one of them beside a list that had just been given one. Every ceiling here now guards a quantity that was enumerated rather than noticed — but the package budget above them still counts the wire, which is why each of the others exists. Distinct counting stores 64-bit hashes rather than strings. By the birthday bound the chance of an accidental collision is about one in 37 million over a million values, and about one in 9 million over the two million the default budget tracks — it rises with the count, not falls. Small enough to call the count exact; not zero, and a collision undercounts. # Performance **This table is the living one.** Measured with the benchmark project: one process per reading, best of two runs — the workbooks on 2026-09-25, the csv files on 2026-09-26, after #9 made csv reading and analysis faster, and the OpenDocument ones (the same two workbooks saved by LibreOffice 26.8) on 2026-09-26. | Fixture | Rows | Cells | Time | Allocated | Per cell | Peak | | ---------------------------------- | --------- | ---------- | ------ | --------- | -------- | ------ | | 72 KB workbook | 10,759 | 17,340 | 0.07 s | 1 MB | 39 B | 57 MB | | 8.6 MB workbook, dense | 100,001 | 1,700,017 | 0.87 s | 61 MB | 38 B | 96 MB | | 101 MB workbook | 1,000,001 | 17,000,017 | 4.6 s | 326 MB | 20 B | 129 MB | | the 8.6 MB workbook as ods (7 MB) | 100,001 | 1,700,017 | 1.7 s | 53 MB | 33 B | 65 MB | | the 101 MB workbook as ods (89 MB) | 1,000,001 | 17,000,017 | 8.7 s | 526 MB | 32 B | 65 MB | | 364 MB csv | 3,000,001 | 51,000,017 | 2.0 s | 1,567 MB | 32 B | 53 MB | | 572 MB csv, malformed | 5,127,969 | 87,175,473 | 3.0 s | 2,742 MB | 33 B | 53 MB | The comparison with other libraries (docs/benchmarks.md) measures the same reads through a different harness and run — 4.30 s for the million-row workbook against 4.6 s here — so quote each table's figures together rather than mixing them. Peak memory stays flat as files grow: the 572 MB csv is read in 53 MB, and a workbook of a million rows in 129 MB. Bytes per cell rises on smaller workbooks because the shared string table is read once and amortised over fewer cells. An OpenDocument spreadsheet reads in less memory than the same workbook — 65 MB for the million rows, since it has no shared string table — and in about twice the time, because it is about four times the XML: the million rows are 1.9 GB of content, against half a gigabyte of worksheet. The sheet names are listed by a pass over that content's bytes before the first row, which is 0.2 s of the 100,000-row file's time. A zipped csv costs its decompression and nothing else. The 572 MB csv zipped to 179 MB reads in 4.2 s against 3.0 s unpacked and analyses in 15.6 s against 14.2 s, the same 49 repairs and the same values to the byte, with a peak of 56 MB against 53 MB (measured 2026-09-26, best of two alternating runs). All of it is measured on .NET 10. On .NET 8 — measured before #9, so the absolute figures below are the older ones; the ratio is what they show — the same code reads and imports the 572 MB csv about 15% slower and analyses it 5–18% slower (import 6.05 s against 5.25–5.32 s; analysis 23.7–23.9 s against 20.1–22.9 s, the widest net10 run being noise), with the same allocations — the runtime's own gains, not a different code path. Behaviour is identical on both; the test suite runs on each. **That table is the reader, not the analysis.** Profiling costs more than reading, and how much more depends on the format: a workbook's numbers and dates arrive already typed and are never parsed, while every value in a csv is text and is tried under each culture in the options. | Fixture | Read | Full analysis | | --------------------------- | ------ | ------------- | | 8.6 MB workbook, 1.7M cells | 0.87 s | 1.5 s | | 101 MB workbook, 17M cells | 4.6 s | 6.7 s | | the same workbook as ods | 8.7 s | 11.6 s | | 364 MB csv, 51M cells | 2.0 s | 9.6 s | | 572 MB csv, 87M cells | 3.0 s | 14.0 s | At ten thousand rows — the size of a typical upload through a mapping screen — analysis is around a third of a second, so none of this is a constraint there. The larger figures are quoted for batch loads, where files are three orders of magnitude larger. ## If analysis ever needs to be faster Two short-circuits are available and neither is free: - **Stop parsing a column once one value fails.** Turns the confidence into a boolean. A column then reports "not all numbers" instead of "99.8%, and here are the 84 that are not", which is the thing the profile exists to say. - **Stop counting distinct values once one repeats.** Turns `DistinctCount` into `IsUnique`. "195 distinct" and "five million distinct" become the same answer. Both are the right trade when a caller only needs the booleans, and both belong behind an option rather than in the default. What is already done, because it costs nothing: a value is checked for whether it could be a number at all before any culture is asked to parse it, which skips the attempt on text columns entirely. [ADR-0001](https://triasdev.github.io/tabular/adr/0001-tabular-parsing-is-our-own-cursor/index.md) holds a **different and deliberately frozen** set: the comparison against every alternative, measured during the spike against the prototype the decision was made on. Its numbers are not these and are not meant to track them. Reproduce this table with the benchmark project and `TABULAR_FIXTURES`. # Benchmarks TriasDev.Tabular against the libraries a .NET developer would otherwise reach for, reading the same large files on the same machine under the same harness. **In short:** on workbooks it is the fastest reader measured, with a peak memory in the same band as the other streaming readers and a fraction of what the object-model libraries need. On clean csv it is third, just behind Sylvan, in a field that is close together. On a malformed csv it is the fastest reader and the only one that reads all 5,127,969 records, where the others merge tens of thousands or millions into their neighbours — and it says what it repaired. ## What the numbers mean - **Time** — wall clock for opening the file and reading every cell of its first sheet. - **Peak memory** — the most memory the process held at any moment (peak resident set). This is the number that decides whether an import fits in the container or server it runs on, and it is the one to compare first. - **Allocated** — everything the garbage collector handed out over the whole run. Memory is reused many times over, so this is far larger than the peak: reading the 5M-row csv allocates about 2.7 GB while never holding more than about 50 MB. It measures pressure on the garbage collector, which shows up in the time, not how much memory a host needs. - **Rows** — rows the library reported, the header included. Where a library reports a different number from TriasDev.Tabular on the same file, the note says by how much. ## How it was measured - Every library does the same work: read the first sheet from start to end and take every cell's value as text, each in the idiom its documentation recommends for speed — Open XML SDK in its streaming (SAX) mode, CsvHelper through its parser rather than record mapping, Sep with unescaping on. - Text, because that is what an import consumes and what every library can hand over. A library that returns typed values pays for formatting them; the bias favours the string-returning readers, TriasDev.Tabular included. - The csv libraries other than TriasDev.Tabular do not detect a dialect, so they are given the delimiter. TriasDev.Tabular detects it. - Each measurement runs in a fresh process — peak memory only ever rises within a process — and is repeated three times; the tables show the median. - Versions are the latest that are free to use commercially: **EPPlus 4.5.3.3** is the last LGPL release (5.0 onwards is under the Polyform Noncommercial licence), and **NPOI 2.7.6** the last under Apache-2.0 (from 2.8.0 the NuGet package ships under a maintenance-fee agreement). Machine: Apple M1 Max, 10 cores, 64 GB, macOS 26.7, .NET 10.0.9, workstation GC. Measured 2026-09-25; TriasDev.Tabular's csv rows re-measured 2026-09-26, after the reader got faster (#9) — its workbook reader did not change. ## Workbooks ### 100k-row workbook — 8.6 MB, 17 columns | Library | Version | Time | Peak memory | Allocated | | ---------------------------- | ------- | ---------- | ----------- | --------- | | **TriasDev.Tabular** | 0.1.0 | **0.81 s** | 96 MB | 61 MB | | Sylvan.Data.Excel | 0.5.8 | 2.11 s | 95 MB | 60 MB | | ExcelDataReader | 3.9.0 | 2.30 s | 92 MB | 456 MB | | EPPlus | 4.5.3.3 | 2.95 s | 502 MB | 1,834 MB | | DocumentFormat.OpenXml (SAX) | 3.5.1 | 3.09 s | 103 MB | 1,359 MB | | MiniExcel | 1.46.0 | 3.88 s | 76 MB | 1,698 MB | | NPOI | 2.7.6 | 5.62 s | 1,747 MB | 2,607 MB | | ClosedXML | 0.105.1 | 5.66 s | 456 MB | 2,501 MB | All read 100,001 rows and the same 1,441,441 values. ### 1M-row workbook — 101 MB, 17 columns | Library | Version | Time | Peak memory | Allocated | | ---------------------------- | ------- | ---------- | ----------- | --------- | | **TriasDev.Tabular** | 0.1.0 | **4.30 s** | 125 MB | 326 MB | | Sylvan.Data.Excel | 0.5.8 | 5.26 s | 126 MB | 324 MB | | ExcelDataReader | 3.9.0 | 9.46 s | 118 MB | 3,381 MB | | EPPlus | 4.5.3.3 | 13.86 s | 1,640 MB | 12,617 MB | | DocumentFormat.OpenXml (SAX) | 3.5.1 | 17.47 s | 134 MB | 10,365 MB | | MiniExcel | 1.46.0 | 19.58 s | 74 MB | 15,100 MB | | ClosedXML | 0.105.1 | 41.71 s | 2,287 MB | 20,957 MB | | NPOI | 2.7.6 | 47.19 s | 11,743 MB | 17,653 MB | All read 1,000,001 rows and the same 14,413,653 values. The streaming readers — TriasDev.Tabular, Sylvan, ExcelDataReader, the Open XML SDK in SAX mode and MiniExcel — hold roughly the same memory whatever the file's size. ClosedXML, EPPlus and NPOI load the workbook into an object model first, which is what makes them good at editing and is why their peak grows with the file: NPOI holds 11.7 GB to read a 101 MB workbook. ## csv ### 3M-row csv — 364 MB, 17 columns, well-formed | Library | Version | Time | Peak memory | Allocated | | -------------------- | ------- | ------ | ----------- | --------- | | Sep | 0.17.1 | 1.12 s | 50 MB | 1,428 MB | | Sylvan.Data.Csv | 1.4.4 | 1.69 s | 52 MB | 1,566 MB | | **TriasDev.Tabular** | 0.1.0 | 1.82 s | 52 MB | 1,567 MB | | CsvHelper | 33.1.0 | 2.19 s | 50 MB | 1,566 MB | All read 3,000,001 rows and the same 43,234,592 values. On clean input Sep is the fastest by a clear margin — it creates no string until one is asked for, where every other reader here makes one per cell; TriasDev.Tabular also spends part of its time on the dialect and encoding detection the others are spared by being told the delimiter. ### 5M-row csv — 572 MB, 17 columns, with malformed quoting A real-world export: quotes inside unquoted fields, text after a closing quote, and 302 quotes that are never closed. | Library | Version | Time | Peak memory | Rows read | Result | | -------------------- | ------- | ---------- | ----------- | ------------- | -------------------------------------------------------------------------------------------------------------------- | | **TriasDev.Tabular** | 0.1.0 | **2.69 s** | **50 MB** | **5,127,969** | every record; 49 stray quotes repaired and reported | | CsvHelper | 33.1.0 | 3.51 s | 71 MB | 5,088,738 | 39,231 records merged into others, no warning — with its default settings too, its bad-data callback is never called | | Sep | 0.17.1 | 4.56 s | 208 MB | 2,845,485 | 2,282,484 records merged into others, no warning | | Sylvan.Data.Csv | 1.4.4 | — | — | — | throws: a delimiter, newline or EOF was expected after a closing quote | **Ground truth.** The file has 5,127,969 lines, and every one of them has exactly 17 fields when split on the delimiter without regard to quotes — so there is one record per line, 5,127,969 including the header. The counts above are measured against that, and TriasDev.Tabular reads all of them. It did not always. An earlier version of this page said "every record" and was checked against our own count rather than against the file: nine lines were joined into six records, each by a field that is a lone quote opening a quoted field and another lone quote in the same column of a later line closing it — valid RFC 4180, so no repair fired. A quoted field that spans lines and holds a whole record's worth of delimiters is now read as the records it is ([#20](https://github.com/TriasDev/tabular/issues/20)); read time is unchanged. This is the file the library's design was decided on (see [ADR-0001](https://triasdev.github.io/tabular/adr/0001-tabular-parsing-is-our-own-cursor/index.md)). A reader that is fast on clean input and silently loses records on dirty input is fast at producing a wrong import. ## Analysis None of the libraries above profiles a file; TriasDev.Tabular does, and it is what the library is for. Analysis reads every row and, for every column, counts empties and distinct values, measures lengths and ranges, tries each value under every configured culture and records where the values that do not fit stand — then ranks type suggestions from those facts. That costs more than reading, and the cost depends on the format: a workbook's numbers and dates arrive already typed, while every csv value is text and is tried under each culture. The *Read* column comes from the library's own benchmark project (one process per reading, best of two), not from the comparison above — which is why the million-row workbook reads in 4.6 s here and 4.30 s there. Different harness, different run; each table is consistent within itself. | File | Rows | Read | Full analysis | Rows per second | Peak memory | | ------------------------- | --------- | ------ | ------------- | --------------- | ----------- | | 100k-row workbook, 8.6 MB | 100,000 | 0.87 s | 1.5 s | 66,000 | 106 MB | | 1M-row workbook, 101 MB | 1,000,000 | 4.6 s | 6.7 s | 149,000 | 188 MB | | 3M-row csv, 364 MB | 3,000,000 | 2.0 s | 9.6 s | 313,000 | 122 MB | | 5M-row csv, 572 MB | 5,127,968 | 3.0 s | 14.0 s | 366,000 | 122 MB | All files have 17 columns. Measured with `benchmarks/TriasDev.Tabular.Benchmarks` on the same machine and day, best of two runs; the peak stays flat because analysis keeps counts and a bounded set of values per column, never the rows. How far analysis could be made faster, and at what price to what it reports, is in the [performance page](https://triasdev.github.io/tabular/performance/#if-analysis-ever-needs-to-be-faster). ## Reproducing The real-world fixtures are not public. `benchmarks/TriasDev.Tabular.FixtureGenerator` writes synthetic files of the same shape — 17 columns, the same row counts, and in the malformed csv the same kinds of quoting defect in similar proportions — deterministically, so every run writes the same bytes: ``` dotnet run -c Release --project benchmarks/TriasDev.Tabular.FixtureGenerator -- /tmp/fixtures # optional scale, e.g. 0.1 ``` It writes `workbook-100k.xlsx` (9 MB), `workbook-1m.xlsx` (91 MB), `clean-3m.csv` (338 MB) and `malformed-5m.csv` (578 MB). On them, TriasDev.Tabular alone (the benchmark project below, same machine as above, 2026-09-25): | File | Read | Peak | Full analysis | Peak | | ------------------ | ------ | ----- | ------------- | ------ | | workbook-100k.xlsx | 0.80 s | 65 MB | 1.27 s | 81 MB | | workbook-1m.xlsx | 3.89 s | 66 MB | 7.05 s | 151 MB | | clean-3m.csv | 2.51 s | 52 MB | 11.8 s | 135 MB | | malformed-5m.csv | 5.71 s | 53 MB | 22.2 s | 135 MB | Same shape, not the same bytes, so expect figures close to the real-file tables rather than equal to them. The malformed file has 5,127,969 lines and reads as all 5,127,969 rows, with 320 stray quotes repaired. Point the comparison project at those files, or at a folder of your own: ``` TABULAR_FIXTURES=/path/to/files \ TABULAR_FILES=big.xlsx,big.csv \ TABULAR_RUNS=3 \ dotnet run -c Release --project benchmarks/TriasDev.Tabular.Comparison ``` `TABULAR_READERS` restricts the run to named libraries, `TABULAR_DELIMITER` sets the delimiter the csv libraries are given (default `;`), and `TABULAR_TIMEOUT` the seconds allowed per run (default 900). The output is Markdown, with the machine and every library's version in its header. `benchmarks/TriasDev.Tabular.Benchmarks` is the other benchmark project: TriasDev.Tabular alone, reader against full analysis, with no third-party packages — the quick check after a change. # Known limitations Behaviour of `TriasDev.Tabular` that is wrong at the edges, or looser than it reads, and has not been changed yet — each with what would make it matter, so you can tell whether it affects your files. None of them loses or silently alters data in files an ordinary producer writes; where one could, it says so and links the issue that tracks it. Found a case that belongs here, or one of these that bites you? Open an issue; a real file that hits an entry is the best reason to fix it. ______________________________________________________________________ ## Correctness at the edges ### A stray quote closed at a field boundary within less than a record joins two lines A quote that opens a field and is closed on a later line by a quote followed by a delimiter or a line ending is read as one field spanning lines, as RFC 4180 says — unless the field has by then swallowed a whole record's worth of delimiters, which marks it as a stray quote (#20, #10). A stray quote that happens to be closed at a field boundary before that point keeps the two lines joined: nothing in the text tells it from a genuine multi-line value. In tables narrower than five columns the delimiter test is off, and only the 100-line bound catches a stray quote. **Matters when** a producer writes lone quotes into fields often enough for two of them to meet at a field boundary within one record. Neither the real-world nor the synthetic 5M-row fixture has such a case. ### A `numFmt` inside `` can overwrite a cell format Number formats are collected from anywhere in `styles.xml`. Differential formats — used by conditional formatting — live in `` and carry their own ids. One colliding with a custom id (164 and up) flips a column between numbers and dates. **Matters when** a file uses conditional formatting *and* custom number formats, and the ids collide. Fix by tracking whether the reader is inside ``, as it already does for ``. ### Number formats the reader does not know are dates `IsBuiltInDateFormat` accepts 14–22 and 45–47. The specification also reserves 27–36 and 50–58 for dates in East Asian locales. A cell using one reads as a number. ### `applyNumberFormat="0"` is ignored The attribute says the format is not applied. Honouring it would change whether a styled cell is read as a date. ### `t="b"` accepts only `"1"`, and `date1904` only `"1"` and `"true"` The schema type is `xsd:boolean`, which also permits `"true"`/`"false"` and `"0"`. A writer using the long spelling produces a boolean read as false, or a 1904 workbook read as 1900 — the latter is a four-year error. ### Relationship targets are not percent-decoded Parts are found through the package relationships, and targets are resolved against their folder, `../` included. A target that percent-encodes its name (`sheet%201.xml`) is not decoded, so it is looked up literally and not found. No producer seen writes one. ### A whitespace-only string is dropped one way and kept the other A shared string of spaces without `xml:space="preserve"` is dropped, because that path reads with `XmlReader` and `IgnoreWhitespace`. The same content written inline is kept, because the scanner has no such notion. Same value, two answers, depending only on how the writer chose to store it. ### Rows that repeat a row number are read as successive rows A worksheet that writes every cell in a `` of its own, all carrying `r="1"`, is read as that many rows, each with one cell; LibreOffice joins them into one. Row numbers here never go backwards or repeat, so a message never points at a row twice. Seen once, in one writer's output. Matters if a producer that writes this way turns up among real files. ### An OpenDocument error cell carries the writer's text, not an Excel code LibreOffice writes a formula that failed with its own wording — `Err:502` where Excel would say `#VALUE!` — and recalculates on saving, so a workbook converted to `.ods` can hold different errors, and different values in volatile or unsupported formulas, than the original. Read as written; a caller comparing the two formats sees the writer's differences, not the reader's. ### Flat OpenDocument (`.fods`) and Excel 2003 XML are not read Both are a spreadsheet as one plain XML document, without the zip. A file that opens with an XML declaration is refused as unsupported rather than read as csv. Matters if anyone sends one. ### Chartsheets and hidden sheets are indistinguishable from ordinary ones `SheetInfo` carries a name and an index. A caller cannot tell a hidden sheet, or a chartsheet with no cells, from a sheet the user meant. ______________________________________________________________________ ## Contracts looser than they read ### A number written in exponential notation is read as text Both the profiler and the extractor parse decimals with `NumberStyles.Number`, which does not include `AllowExponent`. Measured against .NET rather than inferred: | value | `NumberStyles.Number` | `NumberStyles.Float` | | ------------ | --------------------- | -------------------- | | `5.4176e-03` | rejected | 0.0054176 | | `1E+5` | rejected | 100000 | | `-2.5e2` | rejected | -250 | The consequence is worse than a refused row, because it happens at analysis too: a column of such values is profiled as text, so a decimal is never proposed for it, and a mapping to a decimal field then fails every row with `value.type-mismatch`. Found on a real export — a column carried `5.4176e-03` — so this is not hypothetical. `NumberStyles.Float` would fix both places, and the reason it has not been changed yet is not the one this entry used to give. `Float` does **not** include `AllowThousands`, which `Number` does — measured, `1,234.56` reads under `Number` and is rejected under `Float`. Losing grouped numbers in a library whose stated subject is telling `1.234,56` from `1,234.56` would be a worse defect than the one being fixed, so the change is `Number | AllowExponent` rather than `Float`, and it has to land in the profiler and the extractor together or a column will be proposed as a decimal the extractor refuses. **Matters when** any file carries scientific notation — scientific instruments, financial exports and anything that has been through a naive `double.ToString()` all do. ### A date must carry two separators, so `15 Jan 2023` is text `DateReading.LooksLikeOne` requires two of `-`, `/`, `.`, or a colon, before a parse is attempted. That is what refuses a decimal (`1.5` reads as the fifth of January under en-US) and a day-and-month (`3/15` takes the current year), and the price is a date written with spaces and a month name. A genuine `0001-01-01` is refused for the neighbouring reason: year one is the marker that the text named no year. **Matters when** a customer exports dates in a long form. The fix is a parse against the culture's year-bearing patterns rather than a shape test, which is more code than the case has so far earned. ### Dates cannot carry a range `MinValue`/`MaxValue` compare numbers, so a range on a date field is refused by the validator (`mapping.constraint-type-mismatch`) rather than silently judged against zero, as it once was. Date ranges — the obvious second use — cannot be expressed yet. **Matters as soon as** a caller wants one. Fix with a date-typed range constraint. ### `CsvCursor.MoveToSheet` does not rewind The interface documents "positions before its first row"; the csv implementation returns `index == 0` and stays where it is. Analysing and then extracting through one cursor instance reads a csv from wherever it stopped. A csv inside an archive does rewind: the archive reopens its file. ### Moving to a sheet cannot be cancelled `ITabularCursor.MoveToSheet` takes no token. For an OpenDocument spreadsheet it reads forward through the content part to the sheet, and for a workbook inside an archive it copies the workbook into memory, so a move can take seconds on a large file. The first move, made while the cursor is opened, does observe the opening's token. Matters if an import must stop within a second while it changes sheets; fixing it means a token on the interface, in a minor release. ### `IsBlank` ignores the binding's empty-equivalents A row whose every mapped cell holds `k.A.` is not skipped. It is counted as produced and handed over as a valid row of entirely absent values. ### `RowError.RawValue` is the trimmed value The spec says the value "as it appeared in the file". Trimming is on by default, so a value failing a length rule *because of* its spaces is reported without them, pointing at something that looks right. ### `ValidateOnly` returns a full-length span of absent values Indistinguishable from a row where every field was legitimately empty. ### `MappedValue` accessors throw, and say nothing about it `Date` on a decimal value throws; `Integer` on a large one throws. `RawCell` documents exactly this hazard for the identical design. `MappedValue.Absent` also reports `Type == Text`, so an absent value is indistinguishable from an absent text value. ### `MappedValue.FromDate` loses the time of day in `Text` Every length, pattern and allowed-value constraint sees `Text`, so a datetime is validated against a date-only string while `Date` still carries the ticks. ### `CsvDialect` lets a caller claim a value was detected `EncodingSource` and `DelimiterSource` are `required` on a record a caller constructs to *override* detection. The provenance exists so a UI can tell a fact from a guess; a hand-built dialect can lie about it. ### A byte order mark overrides a caller-specified encoding `StreamReader` is constructed with `detectEncodingFromByteOrderMarks: true` alongside the chosen encoding. ### The dialect override is all or nothing A caller who knows only the delimiter must also supply the encoding, losing detection for it. There is no per-property override, no line-ending member, and the quote character is hard-coded with no provenance. ### The package ceiling counts bytes off the wire, not memory `MaxUncompressedBytes` sums the entries' declared sizes. Text decoded to UTF-16 doubles, and a buffer that grows to hold a token peaks at three times its content. The scanner buffer and the csv field have their own ceilings, which carry most of this weight, but the package ceiling by itself is not the bound it appears to be. ### A zip entry that understates its size is read short, without an error The runtime's zip reader stops an entry at the size its headers declare and does not check the checksum, so a crafted archive — or a buggy writer — that declares 10 bytes for a csv of a hundred thousand rows yields its first rows as a plausible, shorter table. Workbook parts are read the same way. An upload cut off in transit is not this case: it loses the directory at the end and is refused as damaged. Catching it needs a CRC-32 over every byte read, which the base class library offers on no x86 processor, and a table-driven one costs about half a second on the 572 MB csv. Matters if files come from a writer that gets sizes wrong. ### A pattern constraint is bounded per value, not per run The match timeout is 100 ms. A pattern using a lookaround falls back to the backtracking engine, and a million rows at 100 ms each is a run measured in hours. Compilation is not bounded at all: a pattern with large counted quantifiers costs time and memory before any value is seen. **Matters when** patterns become admin-authored or config-driven rather than domain-authored. ### Per-column allowances multiply by the column count Each column keeps ten first values, sixty-four frequency keys and up to a hundred and twenty located outliers, each an unbounded string. The column count is now bounded by the format's last column, so the product is bounded — but it is 16,384 times those allowances. ### A workbook's own strings have no length ceiling Sheet names, relationship targets and number-format codes are taken straight from the package with a ceiling on how many there may be and none on how long each may be. Measured: 4,096 sheets — exactly the permitted number — with 400,000-character names is a 1.65 MB upload that retains 3,125 MB for the cursor's whole life. This is the third appearance of one mistake: **a ceiling on the number of things rather than on their size.** It was fixed for the shared string table, which has both, and left standing on its three siblings. **Matters when** somebody sends a file built to do this. An ordinary export cannot reach it. ### A sheet whose part is missing aborts the analysis `MoveToSheet` throws rather than returning false, so a workbook declaring a sheet whose part is absent fails the whole pass instead of skipping it. ______________________________________________________________________ ## Deliberate choices Decided, not deferred — listed so they are not mistaken for gaps. - **Synchronous throughout.** Parsing is processor work over a buffered stream, and a row cannot be a `ReadOnlySpan` and be awaited at once. See the remarks on `ITabularCursor`. - **No comment syntax in csv.** The format does not define one. Add it if the files we receive use it. - **The header is the first row.** No heuristic looks elsewhere; `MappingPlan.HeaderRowIndex` is where a user says otherwise. - **`"C" Road` is repaired without a diagnostic.** The repair is lossy, and unlike an unterminated quote it is not counted. Whether it should be is a judgement, not an oversight.