Why is Smart Clean All slower than individual operations?

Large files are processed in a streaming pipeline rather than loaded into memory all at once. This is what allows the Data Cleaner to finish on multi-gigabyte files in a single browser tab — the file is read, transformed, and written in chunks, so memory usage stays bounded regardless of how large the input is.

How does duplicate detection scale with file size?

Duplicate removal uses a streaming external merge-sort. In the first pass, a compact sort key for each row is written to the browser's private file system (OPFS) — never the full row. The second pass merges those keys to identify the first occurrence of each unique row. The third pass streams the original file again and writes only the kept rows, in their original order. Because only sort keys are held in memory rather than every row, deduplication completes on files too large to fit in RAM — where a traditional in-memory approach would run out of memory and crash.

Engine v2.3 · OPFS streaming · Benchmarked May 2026 · dev machine (Alder Lake class) · Chrome · Windows 11

Data Cleaner Performance Benchmarks

Data Cleaner runs entirely in your browser on a disk-backed OPFS streaming engine. Deduplicating 10 million rows completes in about 88 seconds, with peak memory held near 200 MB. Here is the methodology, the memory model, and what affects speed on your machine.

How it scales: Deduplication uses a disk-backed OPFS 3-pass engine, so memory stays bounded (~200 MB at 10 million rows) instead of loading the whole file into RAM. A 10-million-row dedup completes in about 88 seconds on the reference machine (no-duplicate baseline; files with real duplicates run slightly longer). Per-operation throughput benchmarks for the other cleaning operations are in progress. Results vary by hardware, browser, and file complexity.

Test Configuration

Hardware & Software

CPU	dev machine (Alder Lake class)
RAM	64GB RAM (dual-channel)
Storage	NVMe SSD
OS	Windows 11 Pro (22H2)
Browser	Chrome (stable), single tab, extensions disabled
DevTools	Closed during all tests (no observer overhead)

Test File Specifications

Row count	100K · 1M · 10M (three separate files)
Columns	15 columns (mix of text, numeric, date, email)
Data type split	40% text, 30% numeric, 20% date, 10% email
File size	1M rows: ~33 MB · 10M rows: ~796 MB
Encoding	UTF-8, comma-delimited, CRLF line endings
Duplicates	Dedup baseline uses a no-duplicate fixture; files with real duplicates run slightly longer
NBSP injected	~5% of text cells contain non-breaking spaces
Empty rows/cols	~3% empty rows, 2 fully empty columns injected

Methodology: Deduplication timings captured from a Node harness with an OPFS shim (May 2026); the 10 GB capability was smoke-tested manually in Chrome with clean output confirmed. Time measured from operation start to completion via performance.now() in the Web Worker.

What the Results Look Like

[Screenshot: Smart Clean All progress bar — 10M row file loading in Chrome. Shows "Processing 10,000,000 rows..." progress overlay with percentage counter and animated spinner. Browser tab title updating live.]

Smart Clean All progress overlay — UI stays fully responsive during processing

[Screenshot: Data Cleaner completion toast — cleaning summary showing rows removed, time elapsed, and remaining row count, with a prominent Download button.]

Completion summary: rows removed, time elapsed, download prompt

[Screenshot: Chrome DevTools Network panel during active cleaning — zero requests in-flight. Filter: All. No XHR/Fetch/WebSocket. Status bar reads "0 requests". Confirms client-side-only processing.]

DevTools Network tab during cleaning: zero network requests — data never leaves browser

[Screenshot: Chrome DevTools Performance timeline — Main Thread idle (green). Web Worker thread active (purple/yellow bursts). Worker handles all processing while main thread stays free for UI interaction.]

Web Worker architecture: heavy processing on background thread, UI thread stays free

[GIF: Deduplication — 10M row file. Drag and drop, OPFS streaming progress, completion toast, cleaned result, Download CSV button. Chrome, Windows 11.]

[GIF: Dedupe by Columns — Column picker opens, Email checkbox selected, Apply, duplicate-emails-removed toast. Real-time before/after chip display.]

[GIF: Advanced Filter + AND/OR — Add "Status = Active" filter chip, toggle AND, add "Revenue > 50000" filter chip, row count updates in real-time from 1M → 127K rows. Export filtered result.]

Memory Usage Benchmark

Peak Memory — Remove Duplicates (OPFS streaming)

1M rows (33 MB file)86 MB

Node harness with OPFS shim, May 2026

10M rows (796 MB file)200 MB

Node harness with OPFS shim, May 2026. Data grew 24× — memory grew only ~2.3×.

Data grew 24× (1M → 10M rows) while peak memory grew only ~2.3× — streaming keeps RAM bounded. Benchmark: Node harness with OPFS, May 2026. Real-browser figures may vary; the 10GB capability was verified in Chrome. Per-operation timing benchmarks are in progress and will be published in a future update.

Scalability Across File Sizes

Rows	File Size	Operation	Peak Memory	Path
1,000,000	33 MB	Remove Duplicates	86 MB	OPFS streaming
10,000,000	796 MB	Remove Duplicates	200 MB	OPFS streaming

Data Cleaner vs Alternatives (10M Rows, Deduplication)

Context matters. These numbers show what alternatives take for a comparable deduplication pass over a 10M row file.

Tool	10M Row Dedup Pass	Setup / Learning Curve	Upload Required	NBSP Detection	Regex Filtering
★ Data Cleaner	~88 seconds	Zero — open browser, drop file	Never	Auto-detected	Visual builder + templates
Excel (formulas)	Crashes above ~1M rows	0 (already installed)	Never	TRIM() misses NBSP	VBA only
OpenRefine	4–8 min (Java app, local)	15–30 min install + learning	Never (local Java app)	Yes (GREL trim)	GREL expressions
Python (pandas)	45–90 sec (depends on script)	Hours (Python + pandas + script)	Never (local script)	Yes (str.strip + regex)	Full regex (str.match)
Cloud CSV tools	30–180 sec + upload time	Minutes (sign up, learn UI)	Yes — file leaves device	Varies by tool	Varies by tool

Data Cleaner figure is the OPFS deduplication benchmark (May 2026). Excel and OpenRefine timings are from community benchmarks and internal testing. Python timing assumes a well-written script with vectorized pandas operations on 10M rows, 15 columns. Cloud tool timing includes file upload at 50 Mbps. Results vary by hardware, browser, and file complexity.

When Data Cleaner Is Slower Than Expected

2–5× slower on low-memory machines

Low-RAM machine (under 8GB available)

Why: Deduplication streams through OPFS and stays memory-bounded, but parsing and result rendering still use RAM. On machines with 8 GB total RAM and other apps running, the OS may start swapping. Symptom: progress stalls.

Fix: Close other browser tabs and apps. Try with a 1M row split first using the CSV Splitter tool.

2–8× slower filter speed

Regex filter with backtracking patterns

Why: Poorly formed regex like /.*(a+)+b/ can cause catastrophic backtracking — exponential time on large text fields. Safe patterns like email validation or ^prefix are fast.

Fix: Use the built-in regex templates (email, phone, URL, ZIP) which are pre-optimized. Test regex on 10K rows first before applying to 10M.

1.5–3× slower per column

Wide files (100+ columns)

Why: Trim whitespace and case standardize iterate over all columns. A file with 150 columns requires 10× more string operations than one with 15 columns. Cleaning time scales roughly linearly with column count.

Fix: Use Column Operations to select/remove columns first, then clean the reduced-column file.

1.5–2.5× slower than Chrome

Safari (macOS / iOS)

Why: Safari's JavaScript engine (JavaScriptCore) has lower Web Worker throughput than Chrome's V8 on CPU-bound string operations. PapaParse streaming parse is also slower on Safari due to different FileReader implementation.

Fix: For best performance on macOS, use Chrome or Firefox. Results vary by Safari version — newer versions (17+) are faster.

Slower than CSV export

Excel (.xlsx) export for large files

Why: Excel export builds an XML-based .xlsx ZIP archive in memory; CSV export is a direct string stream and is always faster.

Fix: Export as CSV first for processing speed. Convert to Excel after using the Excel Converter if your workflow requires .xlsx.

1.2–2× slower than single filter

Filtering with AND logic and 5+ active filters

Why: AND filter applies each filter predicate in sequence — the dataset is scanned up to N times (once per filter). With 5 filters, this is 5 full passes over the dataset. OR logic is slightly faster because early exits are possible.

Fix: Reduce active filters to the minimum needed. Remove filters that do not meaningfully narrow results. Regex filters are the most expensive — place them last.

Calculate Your Annual Savings

ROI Calculator — Your Exact Savings

CSV/Excel files cleaned per week

Average rows per file

500K

Manual cleaning time per file (minutes)

45 min

Your hourly rate (USD)

$60

Manual hours/year

117 hrs

7020 total minutes

With Data Cleaner

1.3 hrs

seconds per file, in-browser

Annual savings

$6,942

116 hours reclaimed

Estimates based on browser-side cleaning with Data Cleaner. Actual savings depend on file size, complexity, and which operations you use. Deduplication benchmark: dev machine (Alder Lake class), 64 GB RAM, Chrome, Windows 11, May 2026.

Full Test Methodology

Test Procedure

1Generate test CSV files using Python script (reproducible random seed). Inject known % of duplicates, NBSP, and empty rows/columns.
2Open Chrome fresh instance, disable all extensions, close DevTools.
3Load splitforge.app/tools/data-cleaner in a single tab.
4Drop test file into the tool. Wait for "Parse complete" signal.
5Click the operation button. Note start time via performance.now() logged in Worker.
6Wait for "complete" message. Record wall-clock time from Worker log.
7Repeat 10 times for each operation. Discard highest and lowest values.
8Average the remaining 8 values. Round to 1 decimal.
9Verify result row count against expected (known % duplicates/empty rows).
10Re-test after Chrome update if version changes.

Reproducibility

This benchmark is independently reproducible. The deduplication benchmark uses a Node harness with an OPFS shim and reproducible fixtures (1M and 10M sequential-unique rows, seed 42). On comparable hardware, a 10-million-row dedup should complete in roughly 80–95 seconds. Deviations well outside that range on similar hardware may indicate a performance regression.

Test file generation: Python script with a fixed random seed (42) generates reproducible fixtures (1M and 10M rows). Test files are available upon request — contact via the SplitForge site.

Timing precision: Times measured via performance.now() posted from the Web Worker at operation start and operation complete. Precision: sub-millisecond. Reported to nearest 0.1 second.

What's included in timing: For "Smart Clean All" — includes parse time. For individual operations — does not include parse time (data already in memory). Export times — from button click to download dialog appearing.

Machine state during tests: No other browser tabs open. No other applications using significant CPU. System idle for 30 seconds before each test session. Tests run at ambient temperature (not during thermal throttle).

Disclaimer: Results vary by hardware, browser version, OS, available RAM, and data complexity. Wide files (100+ columns), deeply nested data, or files with many formula-like values may be slower. Mobile results typically 3–5× slower than the test hardware.

Benchmark Changelog

v2.3May 2026

Added NBSP detection to Trim Whitespace — catches 5–12% more whitespace issues.
Deduplication migrated to a disk-backed OPFS 3-pass engine (was an in-memory Set) — peak memory now bounded near 200 MB at 10M rows, enabling files up to 10 GB. Legacy in-memory throughput figures no longer apply.
Added per-column case transform — no measurable performance change vs all-column transform
Column picker for Replace Empty Values: negligible additional overhead (<0.1s)

v2.2November 2025

Upgrade to PapaParse 5.3.2 from 5.3.0 — ~8% faster streaming parse on Chrome
Dedupe algorithm changed from Array.sort to Set-based hashing (later superseded by the OPFS 3-pass engine in v2.3).
Regex filter: pre-compile RegExp objects on filter apply (was re-compiling per row) — 3× faster regex filtering

Honest Limitations: Where Data Cleaner Falls Short

No tool is perfect for every use case. Here's where Excel might be a better choice, and the real limitations of our browser-based architecture.

Browser-Based Processing

Performance depends on your device's RAM and CPU. Modern laptops (2022+) handle 10M+ rows easily, but older devices may struggle with very large files.

Workaround:

Close unnecessary browser tabs to free up memory. For files over 50M rows, consider database solutions.

No Offline Mode (Initial Load)

Requires internet connection to load the tool initially. Processing happens offline in your browser after loading.

Workaround:

Once loaded, you can disconnect and continue processing. For true offline environments, desktop tools may be better.

Browser Tab Memory Limits

Most browsers limit individual tabs to 2-4GB RAM. This is the practical ceiling for file size.

Workaround:

Use 64-bit browsers with sufficient RAM. Chrome and Firefox handle large files best.

No Formulas or Computed Columns

Cleans existing values — it does not compute new ones. There are no formulas or calculated columns; for derived data use Excel or the VLOOKUP & Join tool.

Workaround:

Use Excel or the VLOOKUP & Join tool for derived data and calculated columns.

Flat Data Only — Formatting Not Preserved

Operates on flat tabular data. Cell formatting, colors, and multi-sheet workbook relationships are not preserved — values are extracted and cleaned, styling is dropped.

Workaround:

Export from Excel to CSV before cleaning, then re-apply formatting after if needed.

Exact-Match Deduplication Only

Deduplication is exact-match on whole rows or selected columns. Fuzzy or approximate matching (for example, treating 'Jon' and 'John' as the same) is not supported.

Workaround:

For fuzzy deduplication, use Python with pandas (fuzzywuzzy / rapidfuzz) or a dedicated data quality platform.

Very Wide Files Can Pressure Memory

Extremely wide files — thousands of columns — can still pressure memory, since each row's full column structure is held during streaming. Tall files (many rows, few columns) scale far better than wide ones.

Workaround:

Split very wide files into narrower column subsets before cleaning, or remove unused columns first. Note: the 2-4GB browser tab RAM limit applies to in-memory operations; OPFS-based deduplication uses disk space instead of RAM, which is why 10GB files complete successfully.

When to Use Excel Instead

You need formulas, pivot tables, or charts

Excel is built for computation and analysis, not bulk cleaning.

💡 Use Excel when your workflow involves formulas, pivot tables, or visual data exploration.

You're exploring data interactively rather than applying repeatable cleaning operations

Bulk repeatable cleaning is SplitForge's strength; ad-hoc exploration is better in a spreadsheet.

💡 Use Excel for interactive exploration of smaller datasets.

Your file fits comfortably in Excel (roughly under 100,000 rows)

For small files, the familiar spreadsheet interface may be faster for one-off tasks.

💡 Use Excel for small files where you prefer a point-and-click spreadsheet interface.

You need fuzzy matching, scripted pipelines, or joins across multiple datasets

Python with pandas is the stronger fit for complex data engineering workflows.

💡 Use Python with pandas for fuzzy matching, scripted pipelines, or multi-dataset joins.

Questions about limitations? Check our FAQ section below or contact us via the feedback button.

Known Limitations

Memory model: bounded by OPFS streaming

Because deduplication streams through OPFS in passes, peak memory stays near 200 MB even at 10 million rows — files up to 10 GB have been smoke-tested in Chrome. Very wide files (many hundreds of columns) increase per-row work; if you hit a ceiling, use the CSV Splitter to process in chunks.

Excel export slowdown on large files

Excel (.xlsx) export builds an XML/ZIP archive and is slower than CSV export, which is a direct stream. For very large files, export as CSV and convert separately.

No fuzzy/phonetic deduplication

Duplicate detection uses exact string matching (after optional case normalization). "Jon Smith" and "John Smith" are treated as different records. For fuzzy deduplication, use the dedicated Remove Duplicates tool with fuzzy matching mode.

Mobile performance (tablets, phones)

Mobile CPUs process string operations 3–5× slower than desktop CPUs. Safari iOS has additional Web Worker limitations. For files over 100K rows, desktop is recommended. Mobile works well for files under 50K rows.

Performance FAQs

How does Smart Clean All combine operations?

How does duplicate detection scale — is it O(n)?

How is my file read? Is it loaded into memory all at once?

What is the maximum file size Data Cleaner can handle?

Does the UI freeze while cleaning large files?

See These Speeds on Your Own Files

Drop your messy CSV into Data Cleaner. No signup, no upload, no wait.

Max validated

10 GB (Chrome)

Dedup, 10M rows

~88 seconds

Peak memory

~200 MB

dev machine (Alder Lake class), 64 GB RAM, Chrome, Windows 11, May 2026. Deduplication benchmark; other operation benchmarks in progress. Results vary by hardware, browser, and file complexity.