r/PostgreSQL • • 11d ago

Community Postgres Conference 2027 CFP is open (San Jose, April 21-23)

Post image
30 Upvotes

Mod hat off for a moment:

I'm one of the organizers of Postgres Conference, and the Call for Proposals for 2027 is now open.

Next year is the conference's 20th, so we want the program to reflect what the community has actually learned running Postgres in the real world.

When/where: April 21-23, 2027, San Jose Marriott, San Jose, CA

This year’s program brings back our core technical tracks: Essentials, Dev, Ops, and Extensions, now joined by a new AI track.

We’re also continuing Professional Development & Wellness and Life & Fun. These newer tracks have been so well received that they’ve become an essential part of the conference experience.

We're especially interested in talks from people running Postgres in production, such as:

- Lessons from production incidents: what broke, how you found it, and what you changed afterward
- Autovacuum and bloat tuning on large or high-churn tables
- HA and replication setups, including failovers that didn't go to plan
- Building extensions, or running them at scale
- Benchmarks with methodology you're willing to share and defend

You don't need to be a known speaker. A clear story about a real problem is exactly what we're looking for, and first-time speakers are welcome.

CFP and details: https://postgresconf.org/conferences/2027

Happy to answer any questions about submitting here. 

What's a Postgres problem you wish someone would give a talk about?


r/PostgreSQL • • 3h ago

Tools pg_pinecone: open-source search extension for Postgres, built by Pinecone

8 Upvotes

Disclosure: I work at Pinecone. I'm posting because we'd like your feedback and input on pg_pinecone.

Repo: https://github.com/pinecone-io/pg_pinecone
Blog post: https://www.pinecone.io/blog/introducing-pg-pinecone/

We just released pg_pinecone, an open-source (Apache 2.0) extension based on the search engine Pinecone runs at scale. It works on your existing vector columns, you query with the SQL you're familiar with, and your data stays in Postgres. No API key required.

How it works

How the native engine works, and how it differs from pgvector's HNSW:

  • Vectors are grouped into clusters (IVF), and each row is stored as a RaBitQ code: about one bit per dimension plus a few correction values. A cluster's codes are packed together, so a search reads a few contiguous pages instead of walking a graph.
  • Search probes the nearest clusters, estimates distances from the codes, then reranks a shortlist using the full vectors from your table.
  • Filtering happens during the search. For a filter that matches 3% of rows or fewer (the default threshold), it scans every matching row and scores each with its full vector, so results are exact. For wider filters it searches more clusters and checks the filter inside each one. Post-filtering HNSW can return fewer rows than LIMIT on narrow filters.

Performance3

Against HNSW (pgvector's HNSW index using halfvec, 1M vectors at 768 dimensions, Postgres 17):

  • Index size: 136 MB vs 2.0 GB
  • 0.95 recall: 1.27 ms vs 1.71 ms. 0.99 recall: 2.3 ms vs 4.7 ms
  • 1% of rows match the filter: 0.981 recall at 8.3 ms vs 0.924 at 56 ms
  • Index build: 45s vs 62s

pg_pinecone’s native engine needs no account, API key, or network. An optional Pinecone API key adds a remote engine, BM25 search, embed(), and rerank().


r/PostgreSQL • • 3h ago

Projects Egres: A Postgres-compatible database for heavy, multi-tenant workloads

4 Upvotes

I love Postgres, and I've spent time fighting the same things in production everyone else has: VACUUM, transaction id wraparound, bloat, a process per connection, uncompressed tables. Most of them trace back to design decisions made decades ago. So I kept coming back to one question: what would Postgres look like if it were designed today?

After a year of work, my answer is Egres, a Postgres-compatible database for heavy workloads and for running thousands of tenants on one server. It speaks the Postgres wire protocol, so psql and your drivers connect unchanged. It is not a fork or an extension: apart from libpg_query for parsing, it shares no code with Postgres. Open source, Apache 2.0.

The post covers the what, the why and the how, including why I didn't build this as a table access method:

https://manishrjain.com/egres/

Before Egres, I founded Dgraph and wrote Badger, so I've been in the database field for a while.

Under the hood is a new storage engine, zdb: a tree of LSM trees, which I call an LSM forest. It runs on local disk or object storage, with block compression and checksums. Garbage collection rides on compactions, so there is no VACUUM and no pg_repack, and timestamps are 64-bit, so nothing wraps.

Indexes hold roaring bitmaps, so a query across two indexed columns is two seeks and one intersection without a composite index. Database or table branching is one manifest update: no data copied, no copy-on-write.

Everything runs on zfiber, a coroutine runtime I wrote for it. CPU ticks, disk IOPS and memory are metered per query and per tenant, so thousands of tenants can share one server without the noisy neighbor problem.

What's not there yet: this release is single node, and query features are very basic: equality-only with no joins. Raft replication is in the code; multi-node clusters come in a future release.

Try it out. I'd love to hear your thoughts on Egres.


r/PostgreSQL • • 4h ago

Tools Pseudogress - high-performance mock tool for PostgreSQL databases

Thumbnail github.com
3 Upvotes

r/PostgreSQL • • 1h ago

Help Me! Bloat and Vacuum Best Practices

• Upvotes

I'm trying to wrap my head around when bloat or vacuum settings need to be addressed. For instance I have a table w/720M rows, and it has 54M dead tuples. So it only has ~7% dead tuples, and autovacuum hasn't happened in a month. However, I feel like why not set the threshold lower, and reduce the dead tuples as close to zero as possible? Am I creating solutions in search of a problem? Is there more info I need to find about the table's usage to determine whether there's value in adjusting the settings?

I have similar questions for bloat. I have a table that's 2.2TB with 286GB of bloat. So it comes out to about 12% bloat which doesn't trigger any alarms. I'm trying to understand the nuance here. If bloat stays steady under a certain percent, is it a non-issue? Is it workload dependent? Since the solutions require table locks, I want to know how to proactively identify if/when bloat needs to be addressed.


r/PostgreSQL • • 7h ago

How-To Passwordless PostgreSQL on Kubernetes with CloudNativePG and workload identity

Thumbnail
0 Upvotes

r/PostgreSQL • • 1d ago

Tools A self-hosted wiki that charts live SQL from Postgres

Thumbnail gallery
18 Upvotes

Hi r/PostgreSQL, I'm David, one of the two developers of Oxynote. My friend Simon and I also maintain ttlcache, a Go caching library with about 1,300 GitHub stars, so this isn't our first open-source project.

I've spent 8 years writing software, 2 of them as an SRE and 3 as a team lead. Every team I've been on had the same documentation problem. The useful stuff lived in someone's head or a private Slack thread, and the wiki was full of stale docs nobody could find.

So we built Oxynote, a self-hosted wiki for engineering teams where a page can hold a live chart that runs its own SQL query against your database each time someone opens it, with the explanation written right next to it. Think "signups per day" or "queue depth" on the same page as the runbook that says what to do when it looks wrong.

We were most careful about letting a wiki query a production database. When you add a connection, Oxynote accepts it only if Postgres already treats it as read-only, through a read-only role or a replica. Otherwise, the connection test fails with "has write access" and asks for a read-only user. Queries also support Grafana-style macros like `$__timeFilter` and `$__timeGroup`, and on pages you mark as reviewable, chart query changes go through a draft and a diff that a teammate approves.

The name comes from oxygen + note: docs that breathe. Live data gives people a reason to open the wiki every day, and docs people open every day stay up to date.

It's Apache-2.0 and runs as a single Docker image: https://github.com/oxynote/oxynote

We started it about a year ago and made it public about 3 months ago. Any feedback is welcome, and I'm happy to answer questions.


r/PostgreSQL • • 9h ago

Tools PostgreSQL Like A Pro: Build, query, and troubleshoot without leaving VS Code

Thumbnail developer.microsoft.com
0 Upvotes

r/PostgreSQL • • 1d ago

Help Me! How are you managing DB architecture for AI agents that need real-time state and historical data?

0 Upvotes

Been looking into how to build an AI agent that maintains a current session state while referencing months of user history, and found out this is basically a classic OLTP vs OLAP problem.

The transactional side (OLTP) needs to instantly record the agent’s current state with low latency, but pulling the historical context is a very different access pattern, which is what OLAP systems are built for. Typically, moving data between these two requires an ETL pipeline, which can mean hours of lag depending on the schedule. If the agent needs to make a decision right now based on past data, waiting on a data warehouse sync doesn't work.

For those building context-heavy AI features, how are you bridging the two?


r/PostgreSQL • • 1d ago

Help Me! Connection inside connection

6 Upvotes

Can i open a new PostgreSQL connection inside another PostgreSQL connection? Rn, i work on on project where i used psycopg for connection. I call the function inside that connection which also starts another connection with database. Is it problem?


r/PostgreSQL • • 2d ago

Help Me! Our databases class was all ER diagrams, now looking for a backend coding course on how an app talks to Postgres

19 Upvotes

Finished my databases class last spring, A on the final, and I can normalize anything you hand me and draw a perfect ER diagram. Then I tried to build a small app against Postgres and was stuck on the basics. Connection pooling, migrations, why my ORM sends 40 queries to render 1 page. I want a backend coding course that covers the app side of the database and im deciding between Boot.dev, Launch School and Pluralsight. What should a course like that cover if its done right?


r/PostgreSQL • • 2d ago

Help Me! My football intelligence app's first beta exposed serious automation problems. I'm rebuilding it with independent workers. What should I watch out for?

Thumbnail
0 Upvotes

Sorry if cross posting isn't allowed but I would love input from people who can help me with making sure I do this correctly. It was a big step for me migrating from SQLite with 1.0 to PostgreSQL for 2.0.

This is my first ever build, don't judge me for using Codex for it but this isn't a 'Build an App in 2 days' kinda thing.

I started building 1.0 in June this year and launched it after 8 weeks to test it in a live environment and allowed it to live and breath for 5-6 weeks before taking it down and deciding to take what I learn't from it to build 2.0 from the ground up.

I really hope to engage with users here and hopefully get a better idea.

Thanks


r/PostgreSQL • • 3d ago

Tools Things logical replication doesn't tell you: what we fixed in PgCache 0.8.0 (DDL, DELETEs under REPLICA IDENTITY DEFAULT, Relation messages)

8 Upvotes

TGIF everybody.

I'm a co-founder of PgCache, a caching proxy for Postgres that keeps cached results fresh from the logical replication stream. We shipped 0.8.0 this week, mostly about details of the logical replication stream that we had to handle better.

If you just want to see whether caching would help your own workload: the Fit Analyzer takes a pg_stat_statements export and runs in your browser (nothing is uploaded). On a phone, here's a sample report that you can look at.

***

Logical replication doesn't carry DDL.

-> PgCache now installs two event triggers (ddl_command_end and sql_drop) that write each schema change into the stream as a logical message inside its own transaction. (note: they stay in place at shutdown, and the shutdown log prints the SQL to remove them)

DELETEs under REPLICA IDENTITY DEFAULT only carry the key. Previously, we were judging whether a DELETE mattered to a filtered IN/NOT IN subquery using that key-only tuple, so every filter column read as NULL.

-> 0.8.0 uses the row's recovered old image, so those deletes now invalidate the way they should.

The Relation message flags the replica identity columns, not the primary key. We were reading them as the primary key, so tables with REPLICA IDENTITY FULL, USING INDEX or NOTHING got re-registered on their first change.

-> The key now comes from the origin catalog.

We also improved covered queries (when a broad query is already in the cache, new queries covered by that data are served straight from cache and never touch origin):

-> a cached users LEFT JOIN orders WHERE already holds every users row that qualifies, so a query on users alone that asks for anything within that gets served from cache automatically.

Fixed a bug: We found that a LIMIT query in earlier versions could cache the wrong rows, because the capped population read dropped the query's ORDER BY.

***

Fun proof: One of our (unpaid) design partners benchmarked 0.8.0 the day after release on a 3-node Kubernetes cluster. On a complex multi-tenant dashboard with 80/20 traffic, PgCache served 24x the requests per second of Postgres alone at 100 tenants, 2.5x at 500, and 1.5x at 1,000. The gap narrows as tenants grow, and at 2,000 one of his three runs came in slower than Postgres alone. Writes are still a weak spot: with 5% writes right after a cold start, that dashboard ran about half as fast as plain Postgres. His full write-up.

PgCache 0.8.0 changelog here. Source-available under Elastic License v2.

Happy to answer any questions! And I'd love to hear from people who have worked on the logical replication stream.


r/PostgreSQL • • 5d ago

Projects From Pinecone to Postgres pgvector

16 Upvotes

We run an image and semantic text search API. The first version used Pinecone for vectors and RDS for everything else and the hardest part was keeping the two in sync and performant

About a year ago we moved everything into Neon with pgvector (Neon wrote up the migration here: Neon blog post). End to end search went from about 200ms to 80ms, mostly from cutting a network hop. The vector query itself takes 3-4ms (Tested on 10M records)

Since our backend is on AWS and Neon uses AWS as well, we cut the network latency down to couple ms

Happy to answer anything about the setup if you're thinking about doing the same


r/PostgreSQL • • 4d ago

Community Undo for Postgres, without losing the rest of your day

Thumbnail hubs.la
0 Upvotes

r/PostgreSQL • • 5d ago

Projects Follow-up: pgJDBC batch rewriting made our 1M-row import 1.7× faster

9 Upvotes

A few days ago I posted a benchmark comparing PostgreSQL COPY, JDBC batching, Scriptella, and MariaDB.

In the comments, u/vincent_sch pointed out that pgJDBC defaults to reWriteBatchedInserts=false, while MariaDB enables its bulk insert optimization by default. Good catch, so I reran the relevant cases with rewriting explicitly enabled and disabled.

Median results for 1 million rows, batch size 1000:

Implementation PostgreSQL off PostgreSQL on MariaDB default
Scriptella 15.995s 9.409s 8.688s
JDBC 11.333s 7.290s 5.571s

Enabling rewriting reduced Scriptella's import time by 41.2% and JDBC's by 35.7%.

The PostgreSQL-MariaDB gap for Scriptella shrank from 7.3 seconds to just 0.7 seconds. MariaDB was still faster in this workload.

Three runs per configuration on the same machine, with the original CSV fixture, batch size, transactions, and correctness checks unchanged. All 18 imports passed verification.

This doesn't change the case for COPY when native bulk loading is appropriate. It does show how much JDBC driver settings can influence a comparison.

Follow-up article with measurements and reproduction details:

https://pvrlabs.xyz/java-performance/experiments/postgresql-batch-rewrite.html


r/PostgreSQL • • 5d ago

Help Me! I almost upgraded my server because of one innocent dropdown

0 Upvotes

A client messaged me saying their order dashboard had become basically unusable. I opened it. Everything worked. They sent me a screen recording. Twelve seconds to load one report.

The app and postgreSQL database were running on a small serverspace VM, so my first thought was that we’d finally outgrown it.

 I was already looking at larger configurations when I checked the slow query log. The report had a date filter. I always tested it with last 30 d. The client had selected all time.

That one dropdown was making postgresql  scan almost two million rows every time someone opened the page. I added the missing composite index and changed the default range. The same report went from twelve seconds to under half a second, and the server stayed exactly the same size

Postgresql wasn’t struggling. I had simply spent months testing the polite version of my own feature.


r/PostgreSQL • • 7d ago

Projects I built a large, realistic sample PostgreSQL database: a fictional retailer's 30-year history (laptop-sized to ~3TB)

92 Upvotes

Most sample databases are either tiny (pagila, Chinook) or a single narrow domain. I wanted one that was realistic, big enough to actually stress a query planner, and about a subject I personally find more interesting. So I ended up building a generator that simulates one company's whole life. (I appreciate that "interesting subject matter" is subjective, so the real audience for this may be small!)

It's Nick's Gaming Emporium, a fictional video-game retailer, 1986 to 2016: founding, the dot-com years, the console boom, decline, liquidation. The company is invented, but the product catalogue is a real game-release list, so titles and release dates are genuine, which makes the sales data feel plausible instead of random.

What's inside:

- Full OLTP: shops, staff, customers, sales, trade-ins, refunds, multi-currency

- A star-schema data warehouse + reporting layer (rollups, reporting procedures)

- A web layer: accounts, reviews, and page views

You don't have to build it yourself. It's available as a ready-to-restore dump in four sizes, so you can match your hardware:

- **tiny / small:** quick to restore on a laptop

- **medium:** a solid single-machine dataset

- **large:** ~3TB restored (a 260GB pg_dump) for real at-scale testing

(The dump ships no planner stats, so ANALYZE after restore. No hard-coded collation, so it restores on any locale.)

It's generated deterministically from a fixed seed, so everyone who builds it gets identical rows. There's also a matching SQL Server build from the same generator, so the same data exists on both engines if you ever want to compare the two.

Free and open source. The schema and indexing are meant to feel like a real system that grew over 30 years, not a textbook-perfect design, so there's plenty to tune or pick apart. Feedback and PRs welcome.

Repo (downloads + restore steps in the README): https://github.com/ukhype83-dev/nicks-gaming-emporium

Transparency note: The generator code was produced with Claude as a pair-programmer.


r/PostgreSQL • • 7d ago

Community The wait is over - anyone want to meet up?

Post image
27 Upvotes

Would be fun to have a redditor meetup 🎉


r/PostgreSQL • • 7d ago

Projects Benchmarking single round trip PostgreSQL RLS at 142,564 reads/sec (4.6us overhead vs unguarded SELECT)

10 Upvotes

Conventional wisdom says enabling PostgreSQL Row Level Security (RLS) adds too much overhead or requires an external cache check (Valkey/Redis) in front of the database.

We benchmarked whether forced RLS could match unguarded reads by eliminating extra network round trips and per row current_setting evaluation:

  • InitPlan policy: USING (tenant_id = (SELECT current_tenant())) with a STABLE function so Postgres evaluates the tenant once per statement as an InitPlan rather than per row.
  • Extended wire pipelining: Sending the set_config tenant binding and the parameterized SELECT in a single pgwire pipeline (one sync, one implicit transaction).

Results on a single node (primbench):

  • 1 worker: Plain unguarded SELECT is 42.9us mean (41.5us p50). Pipelined RLS is 47.5us mean (46.2us p50), adding 4.6us.
  • 64 workers: Pipelined RLS hits 142,564 reads/sec (446us mean), matching unguarded reads (143,190 reads/sec) and running 2x faster than a local Valkey grant check + query path (71,842 ops/sec).

Runnable Go benchmark harness and paper (SPEC.pdf): https://github.com/MOG-Systems-Inc/primbench


r/PostgreSQL • • 8d ago

Community Postgres 19 delayed after AI reviewers find more bugs than humans can fix

Thumbnail theregister.com
470 Upvotes

r/PostgreSQL • • 7d ago

Projects Melange: Compiling OpenFGA authorization into native PL/pgSQL functions (1 year later)

15 Upvotes

Hey everyone,

About a year ago I started working on Melange, an open-source authorization system that runs entirely inside Postgres. I originally shared it on r/golang when it was still experimental. Since then I've spent a lot of time improving the generated SQL, performance, and tooling, so I thought I'd share it here.

The motivation

I like the relationship-based authorization model used by OpenFGA and SpiceDB, but running a separate service and keeping authorization data synchronized with an application database adds overhead that I find hard to justify at smaller scales.

Melange takes a different approach: it compiles OpenFGA schemas into PL/pgSQL functions, with relationship traversals, unions, intersections, and exclusions compiled into SQL rather than interpreted at runtime.

Authorization data stays in your existing domain tables, exposed through a view:

CREATE VIEW melange_tuples AS
SELECT 'user' AS subject_type, user_id::text AS subject_id,
       role AS relation, 'folder' AS object_type, folder_id::text AS object_id
FROM folder_members
UNION ALL
SELECT 'folder', folder_id::text, 'parent', 'document', id::text
FROM documents;

Run melange migrate, and permissions can be checked or queried directly in SQL:

-- Check a permission
SELECT check_permission('user', 'alice', 'viewer', 'document', '42');

-- Query accessible documents
SELECT d.*
FROM documents d
JOIN list_accessible_objects('user', 'alice', 'viewer', 'document', 50, NULL) a
  ON d.id::text = a.object_id;

There's no separate service or tuple synchronization, and checks run inside the same transaction as your application data, including seeing uncommitted writes.

What's changed over the last year

Performance: Listing functions now use set-oriented SQL instead of checking permissions row by row. On one workload of ~90k tuples, list_accessible_objects dropped from ~4 seconds to 12 ms. Some recursive listings improved by over 100x. Individual permission checks benchmark around 130 µs database-side, with relatively flat performance as the dataset grows in tested workloads.

Correctness: The test suite now compares results against a real OpenFGA server, fuzzes generated nested authorization models, and checks listing results against individual permission checks. This caught several bugs, including cases where Melange incorrectly granted access. These are fixed in v0.10.0. Existing users should upgrade and rerun melange migrate.

Debugging: explain_permission() returns a JSONB trace of why access was granted or denied, while expand_permission() shows the granting relationships as a tree. There's also melange studio, a local, read-only UI for inspecting models, debugging checks, and running EXPLAIN ANALYZE.

Migrations: Melange supports dedicated schemas, generating versioned SQL for Flyway/golang-migrate, comparing deployed models for breaking changes, and recommending indexes based on your schema.

Distributing software as SQL

One aspect I find pretty interesting is the deployment model. Melange isn't a PostgreSQL extension. It generates ordinary SQL and PL/pgSQL functions installed directly into the database. No native binaries, extension installation, or separate runtime. Once deployed, the CLI isn't needed.

Effectively, the SQL itself is the distributed software. It runs on standard PostgreSQL 14+ instances, including managed databases like RDS and Cloud SQL, without requiring custom extensions.

I haven't seen many (if any) projects take this approach. It's introduced some interesting challenges around migrations, versioning, schema isolation, and managing third-party code inside someone else's database.

Trying it out

Melange is MIT licensed, supports PostgreSQL 14+, and the CLI is written in Go.

brew install pthm/tap/melange

The project has been growing steadily, with a few people contributing fixes and ideas. Contributions and feedback are always welcome.


r/PostgreSQL • • 8d ago

Tools PostgreSQL COPY vs batched JDBC: importing 1 million CSV rows

13 Upvotes

I maintain Scriptella, an open-source Java ETL tool, and recently ran an experiment comparing different approaches to importing 1 million CSV rows into PostgreSQL and MariaDB.

The comparison included PostgreSQL COPY, handwritten JDBC, Scriptella, and Python's csvsql.

A few interesting results from the local PostgreSQL runs:

  • Native COPY: ~1.7 seconds
  • Batched JDBC: ~13.6 seconds
  • Batched Scriptella: ~14.8 seconds

Unsurprisingly, PostgreSQL COPY was by far the fastest for a straightforward CSV import. Batching also made an enormous difference compared with individual inserts.

Interestingly, MariaDB was faster than PostgreSQL for the batched JDBC and Scriptella cases in this setup, although this isn't a general database performance comparison.

Of course, raw import speed isn't the only reason to use ETL tools. Transformations, multiple data sources, and reusable workflows can justify some overhead. For a direct CSV import into PostgreSQL, though, COPY is an excellent choice.

The experiment is exploratory, with limited repetitions, but includes reproducible code, raw measurements, and correctness verification.

Article with charts and analysis:
https://pvrlabs.xyz/articles/importing-a-million-csv-rows.html

Benchmark source and reproduction instructions:
https://github.com/scriptella/scriptella-benchmarks

Curious how others approach bulk imports in PostgreSQL. Do you typically start with COPY, or use JDBC batching when additional processing is involved?


r/PostgreSQL • • 8d ago

Tools Should a migration linter say "SAFE" when it couldn't run a safety check?

0 Upvotes

I released "safe-migrate v0.11.0" today. One of the changes I spent time thinking about was how the report handles checks it couldn't run.

The tool simulates migrations against a synchronized PostgreSQL catalog snapshot. If the snapshot only covers selected schemas, an object missing from that snapshot isn't necessarily missing from the actual database. A check that depends on it may simply be impossible to evaluate.

In v0.11.0, those skipped checks are listed under "not_evaluated", with the reason and any missing schemas the tool could identify. If a skipped check could have produced a Tier 1 finding, the overall verdict is capped at "CAUTIOUS" instead of reporting "SAFE".

I chose to list the skipped checks rather than add a separate "UNKNOWN" verdict. I'm not sure which approach makes more sense for CI: keep the existing verdicts and explain what wasn't checked, or have a distinct unknown state?

For context, safe-migrate is a Rust CLI that analyzes PostgreSQL migrations against a database snapshot. It never executes the migration SQL.

"Release and binaries" "Changelog"

One breaking change for existing users: the cache format is now V10, so run "safe-migrate sync" after upgrading. The JSON report schema is also now v3.


r/PostgreSQL • • 8d ago

Community PostgreSQL extensibility at WAWTech — Warsaw, 26–27 November

Thumbnail
2 Upvotes