Submit your salary here

I have been working for 5 years on the analytical side of things and I'm yet to find a company where our work actually makes a difference. I know that there are some very specific businesses like financial trading where profit is actually made using data, but for most corporate BI projects this is not the case.

Companies have huge budgets for their data teams to support business decisions, yet most of our output is never used to make a difference. Business seems to care more about a KPI 'looking good' than being actually correct, if they care at all.

I have worked in very different environments but this was all the same everywhere.

I am now working in digital advertising which is supposed to be heavily data driven and we produce more than 30 different KPIs yet advertisers don't seem to care and just spend their yearly advertising budgets anyway.

When I was at uni, 'big data' was hyped as the holy grail, and I was naive enough to think that my work would actually end up in board meetings and that we would be one of the most important departments in the company.

Can someone more experienced tell me why this is?

We're a construction company using Viewpoint Spectrum as our primary data source. Currently, most reporting is done through Power BI, SQL, and Claude, with approximately 20-40 report consumers and a small number of report developers (2-5).

One challenge is that Spectrum database access is restricted through IP whitelisting, making direct access difficult for field users. Additionally, with Claude being use a lot I believe it's important to have real data modeling and etc. We're evaluating different architectures for reporting, governance, and future AI use cases.

I looked at data warehouses and other solutions be they seem to be very expensive and hard to justify. I have also seen people recommend having your own server where something like MongoDB or Apache Superset is hosted. There are too many solutions so I would like some recommendations.

I am a relatively new BI Manager. When I inherited this setup, the data engineers were using the AWS CDK to deploy materialized views and views in our BI Postgres database which we use as a data warehouse. From my research, the BI team should be using dbt to manage these data objects. When I asked the CTO about this, he said he'd rather not introduce new tools (dbt and AirFlow) when the current setup is working. For what I know, the CDK deployment via github can be automated for views, but not for materialized views because materialized views need to be manually dropped and created. I don't have enough knowledge to tell whether the current setup is 'working' or not. Would dbt + AirFlow work better? How could I articulate the advantages of dbt over the AWS CDK?

Not sure if relevant, but we also plan to switch to a data warehouse system (Snowflake, Redshift etc.) at some point.

I used to build data pipelines. I deeply enjoyed ETL/ELT work and now most of that is done using AI. Now my work has shifted to building semantic layers and it’s so boring.

I admit it.

I just spent weeks tidying up our Databricks access layer. I created entra groups for sensitive schemas, terraform to deploy them and set the entitlement and grants for admins, engineers, and default users.

My technical colleagues would ask what I was up to, and when I told them they would sigh with empathy - “access control, rough - super boring right?”. Their expression turn to horror on realising I created this work for myself, because I wanted to.

I found it immensely satisfying taking a mess of permissions, scattered across repos, one-off manual grants, etc and building a fresh, new access control model. I learnt about the different ACL levels in databricks and ultimately ended up terraforming everything after initially setting up Jobs in unity catalog, before learning of its limitations granting permissions to workspace objects.

Now everything is managed by IT - want access to sensitive data? Raise a ticket. No more coming to a dev and asking to be manually added to the workspace.
It’s a relatively standard implementation, but I’m proud of it :)
I think one of my favorite things about tech is ‘tidying up’ - whether it be modularising code, creating libraries and generally just turning a mess into something simple and intuitive.

(Probably neurodivergent btw yet to be tested)

Edit: spelling

While it is really easy to create semantic views with AI, it is more difficult to ensure that they answer correctly, that they are maintained over time, and that errors and drifts are actually caught.

I created this system where I have AI help me with the initial creation of the context that goes in the semantic views. I also created me canary jobs utilizing Snowflake tasks and alerts to ensure that, if the data drifts over time for specific answers I would expect to stay stable, I'm alerted to those and able to investigate that and thus maintain accurate self-service analytics.

Beyond this approach, I'd love to hear how other people also maintain their semantic views.

I actually want to know if all the companies are solely focusing on building AI agents from snowflake to multiple places or do companies have actual good products out of data team.

I recently started a new role as a Data Governance Manager at a mid size managed care organization. One of my first asks has been to create test templates for the new set of infrastructure tables. Essentially it’s to ensure all the tables that are now in production don’t have a glaring issue or whenever we make a change we go through the testing again.

My background is in data analytics and data management. I have very limited experience as a data engineer so aside from data quality tests I really don’t know what kind of tests should be run. I have used Claude to create a template of standard test cases but they seem either too convoluted or too simple like checking for nulls in non-nullable fields. Any resources or advice on how to ramp up or get a better understanding of this?

I have seen too many departmental only solutions - sutura is my attempt to build something bigger, for the whole enterprise ;) - it is a secure multiplayer chatting with data experience which is pluggable for data and semantics.

- https://github.com/telekom/sutura/

- https://telekom.github.io/sutura/main/slides/2026-introducing-sutura/#/1

Hello. We are using now a custom orchestrator but thinking about moving to a established one. We have an event-driven processing setup, where the landing of files in Google Cloud Storage buckets trigger some dataform and/or Python processes.

We deal with about 2 billion events per month. In some cases it's not an instant trigger of the process, because we can configure a buffer and wait to gather a bunch of events together.

At first, the better suited decision would seem Dagster, but it being aqcuired by Prefect really makes me afraid about migrating and then having to remigrate again in not so long time.

So, how good and scalable are event driven schedules in Airflow? We might be using Managed Airflow in Google Cloud, or hosting our own deployment in Kubernetes. I do like Airflow and have used it in a few companys, but just for periodic workflows.

We have a table design that is fact_opportunityproducts, which is our Salesforce line fact table that has a row for each product per opportunity. But unlike event data our financial data these rows can be updated (opportunity can be closed, amount and quantity can change etc) such that we ofcourse cannot do normal incremental loading. What are design choices to be made? And we cannot only look at the salesforce last update date because we have created fields that also depend on other system inputs.
Do you then do rcompute the complete view but only merges the records that are different based on a hash? But that can also be very costly.

Have any of you done this on a mature project, and was it worth it?

We originally picked dbt Cloud for our greenfield DWH project, attracted by the idea of not having to host the infrastructure and being able to focus on the data instead. With the initial high-growth phase nearing its end, I am now looking at optimising what has been built and could not make a good argument to carry on using Cloud.

The repo and CI/CD pipelines are hosted externally. Studio is barely touched as most developers have now switched to VS Code and Claude does all the AI coding we need. Even the semantic layer is managed in an external tool that did not integrate very well with what dbt offers natively.

This pretty much leaves job schedules and monitoring in the Cloud. This looks like terrible value for the money, even though we got a very good deal for the initial year (I imagine the renewal will hit us in the face).

I wonder if anybody ended up in a situation like this? Give me some good reasons not to move to Core.

Hi, So my team decided to sunset Informatica and move all our workflows (atleast 500+) to our config driven ETL framework which is basically a stored procedure reading a config table and running the jobs (a stored procedure again). This already existed for about a year and this where they are putting all their money in. And now they want to move all our informatica jobs to it. So far we've developed the following procedures:

1. Flat file to staging table
2. Staging to silver (removing duplicates, trimming and casting)

After that they started assigning me Informatica jobs to migrate, now these jobs are essentially: 1. Ingest flat file to the staging table. 2. Run post sql stored proc which moves data to silver and so on...

Now I figured the logical and the easiest way to do this was to: 1. Have a config table to keep track of jobs, corresponding metadata and the POST SQL Stored proc run by informatica. 1. Run our flat file to staging procedure. 2. Run the POST SQL stored proc from informatica (configured in the config table) to move the process the data EXACTLY the way we do it currently. 3. Run the Staging to silver SP (we newly developed) to handle duplicates and casting to correct data types.

I pitch this my boss and he did not like it, he was like we doesn't want 500 stored procs polluting our namespace. So instead he suggested putting the POST SQL logic (from Informatica workflow) in the tab_comments table, where we put the column update statements in the comments table:

COMMENT ON COLUMN table_name.column_name IS 'UPDATE table_name set columnA = columnA + 1'; And then iterate through all the column comments and run those statements using dynamic sql.

I do not like this only cuz it would be such a pain to develop and debug. I told my reservations about this and he was like 'it doesn't matter what you think, you'll do what I say'.

Has anyone migrated from Informatica to a pure SP based ETL framework? How did y'all do it? I want to pitch a better idea for the sake of my sanity since I'll be put on this task for a while.

Any info is appreciated, thanks

My team just use ordinary excel files, google sheets as its main source of data for the dashboard. Now, they want the team to be able to contribute and view the data without opening the excel file. So what exactly low cost way I can do to centralize the data so that all team members can view and contribute to it?

That’s where I’m at. Like yeah sure I can take the recruiter call, but given the high supply and limited demand of the market…I have a hard time getting motivated to throw myself into a cycle despite getting nibbles. Especially knowing all the prep it takes on top of the day job.

For the savants who breeze through processes and get whatever job they go for—this post is not directed at you. It’s for the rest of us who are good but not top percentile and still have to compete with everyone else.

Hi,

I created a DE stack (fully integrated services) and learning material for those interested in self learning. I kept Databricks, fabric in mind while building this stack. Feel free to use it and let know if there are any improvements/suggestion I can make

- Work independently in your own data lakehouse.
- Invite other team members to collaborate in your lakehouse.
- Store secrets securely using Key Vault.
- Upload raw files using the built-in file manager.
- Transform raw data into Bronze, Silver, and Gold layers using pre-connected Jupyter notebooks.
- Query your data lakehouse and data warehouse.
- Build BI dashboards using Apache Superset.
- Orchestrate notebooks using Apache Airflow.
- Run notebooks on a distributed Spark cluster.
- Manage multiple teams.
- Authenticate securely using Single Sign-On (SSO).
- Get started with dbt integration.
- Learn at your own pace with self-learning tutorials.
- Manage data governance using the built-in data catalog.

https://maarthala.github.io/data-engineering-lab/

Boomi acquiring Elementary after Rivery. Apologies if this was already posted before. However, I wanted to share a meme. This post is not for discussion. Just the meme. Thanks

I've been working with DuckLake recently, and learned about it's ability to traverse snapshots... So, I decided to create a little UI to allow you to diff snapshots on import, or while working with a table.

It's part of a larger product, but still thought it'd be cool to share here. DuckLake is seriously underrated. Hopefully it gets more traction going forward. I feel like this should be the default for all OLAP systems at this point...

You can do all of the things you expect, including seeing total row or column differences, and cell-differences upon click. It collapses row ranges that weren't affected, just like git clients do. So if you have 3 million rows, but only certain 25 rows changed across 3 ranges, you'll see just those ranges.

If you are focused on a single table, you can also filter to only see the snapshots that directly affected that table, which is useful if you have a ton of commits on other, non-relevant tables. (This was inspired by the LLDB "stack-collapse" where you can filter to which instructions directly affect the thread you're on).

Works with local and Postgres DuckLake catalogs with data stored locally or S3.

Here's the link if you're interested: https://www.cocoalemana.com/integrations/ducklake/

Currently a Senior DE (really more of an AE) at a decently big company in a boring industry with ab 5yoe (27). I work with SQL, Python, have cloud experience, orchestration, dbt, snowflake, standard DE stuff. I want to do more heavy realtime/big data work with kafka/kubernetes/spark but I feel kind of stuck in a loop because recruiters only reach out to me about jobs using sql/python (because thats where I have experience) and I feel like im starting to approach my ceiling with these roles in analytics in terms of pay (I’ll likely max out at 150 whereas in doing actual big data engineering that would be closer to 3/400k and it’s way more interesting to me so I feel like I’d be much better at it). I want to be more technical and do more real DE work as opposed to working in analytics (which ime has just been a lot of batch processing stuff) - is it cope that I’m thinking of doing a masters in CS to acquire the skills I need to make that jump? I did my undergrad in finance and I feel like recruiters often overlook me because of this and go with more accredited candidates (people with cs bachelors or masters) even if I may be a better fit for the job. Am I coping and should I just grind LC and build projects with these tools until I get a new job? Or is a CS MS a viable option? Thinking of applying to GT OMSCS and have heard people have varying rates of success

Do we need to build an additional layer where AI Agents can post questions that layer - like Snowflake Cortex can query and get the response for the agent.

Would love to get views and best practices - is this a big problem?

I know pg_cron is primarily designed to run scheduled maintenance commands on the DB but I was wondering if someone has used it to kick off a batch extraction on your data pipelines.

I have reservations against using pg_cron for this namely the following:

1. The scope of pg_cron in this scenario is outside the bounds of the DB.

2. The boundary of using the DB to store only activity data is violated as it now has some operational data in it.

3. SCM or an additional dash for setting up jobs will be lost if this pattern is followed.

I'm curious to know your thoughts