TL;DR: Our ERP stores everything in AcuCOBOL Vision files. The only official way in was a 3-seat ODBC driver that was always maxed out and occasionally crashed. We reverse engineered the file format, now read the raw .dat files directly (read-only), and mirror 66 tables into Postgres every 5 minutes in about 35 seconds. It took a few weeks, one scary mistake, and a lot of verification.
The setup
We're a small food manufacturer running an ERP that's been around since the 90s. Under the hood it's AcuCOBOL, and every table is a Vision indexed file: a .dat with the records and a .vix with the index.
Everything modern we run (dashboards, EDI, order imports, a wholesale portal) reads a Postgres mirror of the ERP. That mirror used to be fed by a bridge over ODBC, through a vendor driver licensed at 3 concurrent seats. Those seats were spoken for all day by shipping stations and automation, the ODBC server is a single-threaded binary from 2014 that pegs one core, and a full refresh took hours. When it fell over, everything downstream went stale.
The idea
The files are just bytes on a share. If we could decode them ourselves, we wouldn't need a seat at all.
What made it possible: the ODBC product ships its own catalog describing every record layout. For each table it lists every field with its byte offset, length, PIC clause, usage, decimal places and date format. That's basically the COBOL copybooks in queryable form, so we never had to guess a field.
Cracking the file format
Vision files have a header area, then records stored in slots, each with a small header of its own. Where the records start, how big that per-record header is, and the stride between records all vary by file. So instead of hand-deriving 66 of them, we wrote a generator: try candidate layouts, decode a sample, and score how many decoded rows exactly match known-good rows pulled over ODBC. The winner goes into a config, and nothing gets in without validating against live data.
Making it safe to cut over
This is the part that actually took the time. We ran the reader in shadow for weeks next to the old bridge:
- every night, a full row-hash comparison of every table, with both pipelines normalized the same way
- every two hours during business hours, count checks and field-by-field spot checks on the busiest tables
- a gate: 10 consecutive clean business-hours cycles on the order header file before we'd switch
- the reader code and configs are hashed together, so any change resets the streak
What it looks like now
- 66 tables re-checked every 5 minutes, about 35 seconds per sweep
- each record's raw bytes are fingerprinted, and only rows that actually changed get written
- files are opened read-only, and the reader never touches the ODBC service (4 small tables still ride the old path while we finish them)
- a row only gets deleted after it's missing from the source on two consecutive sweeps, so one bad read can't wipe live data
Things that bit us
- Live users lock records. Someone editing an order holds a byte-range lock on part of the file, and any read that hits it fails. We retry with backoff and treat a lock as "no data this pass," not an error. Once a desktop session died mid-edit and its server-side runtime held a lock for five and a half hours.
- Files change under you. We check size and mtime before and after each read and retry if they moved.
- REDEFINES. Two fields describing the same bytes, and the catalog's ordering of the tie isn't something you can derive.
- "Last synced" changed meaning. The old bridge rewrote every row on every pass, so a timestamp meant "last checked." Ours only writes changes, so it means "last changed." Four freshness monitors started calling quiet tables dead, and all of them had to be repointed at the reader's own heartbeat.
- Our own big mistake. During cleanup we deleted about 1.4M rows we believed were stale leftovers. We'd found them by comparing against the reader's output, but the reader deliberately skips very old records, so everything older than that cutoff looked "missing." It was real history. Every deleted row had been copied aside first, so it came back as archive tables. Lesson: an anti-join only tells you something is stale if both sides cover the same key range.
- Bloat. The old bridge rewriting every row every 15 minutes for months left some Postgres tables 7x their real size. One night of compaction took the database from 45 GB to 32 GB.
How it got built
Full disclosure: most of the reverse engineering and code was done pair-programming with Claude. It was great at grinding through byte offsets and writing verification harnesses. It was also the one that made the delete mistake, and the one that dug it back out, which honestly felt about right for a pair programmer.
Happy to answer questions if anyone else is stuck feeding a COBOL system through an ODBC straw.