https://www.ainotai.maketools.ai/

I don't think we should use AI for EVERYTHING ! Some tasks require a simple script, automation or formula. This game is a fun way to explain it :)

A dashboard can be visually clean and still be wrong. Before I trust a reporting dataset, I like to validate the assumptions underneath it—not just whether the query runs.

Here’s a practical checklist:

1. State the grain

Write down what one row represents (for example, one order line, one ticket event, or one daily account snapshot). If two tables have different grains, joining them without accounting for that difference can inflate counts and sums.

2. Test key uniqueness before joining

For every expected one-to-one or many-to-one join, check whether the lookup key is actually unique:

sql

SELECT customer_id, COUNT(*) AS row_count

FROM dim_customer

GROUP BY customer_id

HAVING COUNT(*) > 1;

Then compare row counts and distinct business keys before and after the join. A query returning successfully doesn’t mean the relationship is correct.

3. Reconcile totals to a trusted source

Compare record counts and important measures by a stable slice such as date, status, or region. When totals differ, quantify the gap and trace it to exclusions, late-arriving data, duplicates, or definition changes.

4. Check freshness and completeness

Confirm the latest expected date arrived, required fields are populated, and categories haven’t unexpectedly disappeared. These checks catch silent partial loads that can make a dashboard look plausible.

5. Validate business rules explicitly

Document definitions for metrics such as “active,” “resolved,” or “on time.” Encode the rules in tests or assertions, and keep exceptions visible instead of burying them in a chart filter.

6. Make the checks part of the pipeline

Automated tests, run logs, and failure alerts turn one-time detective work into a repeatable control. The goal is to catch a bad refresh before stakeholders make decisions from it.

My rule of thumb: validate the grain, joins, totals, and freshness before debating the chart. What’s the one check you consider non-negotiable in your analytics workflow?

The project is growing, and we are trying partners capable of delivering big datasets on a steady basis, potentially millions of samples, backed by strong quality checks. I have looked into the major annotation providers, but their websites make it difficult to separate the ones that handle real volume from the ones with strong marketing.

Here is what I want to understand:

Which providers have handled high-volume data collection or labelling for you, and would you hire them again?

What did you do to test quality before signing a large contract? A trial batch, a sample review, or something different?

Which warning signs did you learn to catch early, like late deliveries, uneven labelling, or vague sourcing and consent practices?

Does it make more sense to rely on one major vendor or to divide the work among a few smaller teams?

I am more interested in first-hand experience than lists of vendors. Once we decide, I will post our choice in case it is useful to anyone.

Disclosure: I created and maintain the Africa AI Incident Observatory.
I’m sharing a small, open dataset for people interested in AI safety, governance, and qualitative incident research. It currently contains 19 curated records across nine primary African countries, available as a CSV with 36 fields.
The aim is to make each record inspectable: what was reported, which sources support it, how strong the evidence is, and what remains uncertain.
**What’s included**
Incident dates, date precision, primary country, and other affected countries.
Sector, system or tool, reported harms, and a short incident summary.
Public source links, source types, and AI Incident Database references where available.
Evidence-confidence grades, qualification notes, and caveats.
Separate severity dimensions and scores under a documented project rubric.
**One design decision I’d like feedback on**
Evidence confidence and severity are recorded separately. The seriousness of a reported harm should not, by itself, make the underlying claim more certain.
The project also keeps a separate watchlist for cases where the reviewed evidence does not adequately establish AI involvement. Those cases are excluded from the core incident count.
**Sources and limitations**
Seventeen of the 19 core records reference existing AI Incident Database entries. This project builds on that work and the public sources cited in the records; the contribution is the African-focused compilation, evidence annotations, and documented review structure.
This is a small, selected collection. It is not comprehensive or statistically representative, and country counts should not be used to estimate incident rates or rank countries by AI safety. The confidence and severity annotations are project assessments that readers can inspect and challenge.
**Access**
Repository, methodology, and documentation:
https://github.com/Briella009/Africa-AI-Incident-Observatory
Direct CSV:
https://raw.githubusercontent.com/Briella009/Africa-AI-Incident-Observatory/main/data/incidents.csv
The original compilation, structure, and curator-written summaries are available under CC BY 4.0. Linked third-party material retains its original terms.
What would you change in the schema to make uncertain AI attribution easier to analyse? I’d particularly welcome source-backed corrections, missing cases, or feedback from anyone who has worked with incident datasets.

"So many new tools, I don't know what I need to learn. And I mean, like Snowflake, Microsoft Fabric, Azure Databricks, Python Spark, AWS."

On a call last month, someone listed everything he felt behind on in that one breath.

The day before, someone else had described what a list like that does to you.

"There's so much going on... and then you're just like, you know, you don't even get started."
That second part is the one that costs people years.

They work hard at this.

What they're missing is a reason to pick one thing and leave the rest alone.

So I don't answer with my own list.

I ask which jobs they'd actually take tomorrow, in one industry, at the pay they're after.

Then we pull ten of those postings and count every tool, every time it shows up.

Anything in seven or more of the ten goes on the list.

Anything in one of the ten can wait, and Kafka ends up there more often than people expect.
What comes out is shorter than the list they walked in with, and it came from jobs they want.

On Wednesday, November 4 at 7pm ET, I'm doing this live with ten real postings in a free session called So Many Tools. Where Do You Start?

I'll build the tally on screen, show you what I'd learn first and what I'd skip, then take your questions.

DM me if you want to join the free webinar

Guys, actually i have 80k company database like its name and lei , now my task is to find the revenue,product type and region, i used hermes agent which almost gave nothing to revenue, it doesnt search, and claude ,chatgpt most of them gave nothing as revvenue value,the only solution was to search each company one by one, gemini gave revenue but i think its actually false , i gave data to gemini by 100, is there any other way ?

I recently started a new role as a Data Governance Manager at a mid size managed care organization. One of my first asks has been to create test templates for the new set of infrastructure tables. Essentially it’s to ensure all the tables that are now in production don’t have a glaring issue or whenever we make a change we go through the testing again.

My background is in data analytics and data management. I have very limited experience as a data engineer so aside from data quality tests I really don’t know what kind of tests should be run. I have used Claude to create a template of standard test cases but they seem either too convoluted or too simple like checking for nulls in non-nullable fields. Any resources or advice on how to ramp up or get a better understanding of this?

Can someone guide if practicing practice exam from CDMP.brightspace alone is enough. Or what more I should do?

A new monthly Chrome UX Report dataset is available in Chrome UX Report Dumps:

The 202609 release contains 10 non-cumulative rank buckets with 18,376,044 entries and 95.1 MiB of compressed data. The repository now holds 68 monthly datasets across 585 files: 1,082,630,540 total entries and 5.41 GiB compressed.

These are ranked website lists extracted from Chrome UX Report data, split into non-overlapping rank buckets for practical downloading and processing. Totals across months are not counts of unique websites; the same sites can appear in multiple monthly datasets.

Feedback and feature requests are welcome.

so I have developed over that last month a data engineering toolkit, I took pain points and built apps that adressed them https://www.loom.com/share/58f2b3c68dd74c8e8dacbe9c703698ea this is just a loom that shows how it works. No self promotion, just asking if this is something that users would appreciate.

After completing an analysis, what checks do you perform to ensure that your calculations, aggregations, and conclusions aree correct? Do you independently validate results using SQL, spreadsheets, or another tool?

We are implementing Collibra as part of the data governance initiative and honestly I’m struggling to see the end of the tunnel.

The tool seems huge and overwhelming: business glossary, document data model, mapping relationships feels like an enormous amount of work that could go forever.

I’d love to hear from people that have Collibra in the companies they work for:
\\- where are you in implementation? Have you reached a point where collibra genuinely is part of your day?
\\- are you getting any value from it?
\\- what are you documenting?

Let me know your honest opinions!

Hey all, welcome. Quick intro since this place is brand new.

I run Sanctus Data Labs. We help business owners sell their operating data to AI companies. I started this sub because the questions I get on calls are the same ones I see asked (badly answered) all over Reddit, and there's no good place to talk about this market honestly. So here's the primer.

What are AI companies actually buying? Who's buying? What's it worth? What do we say when we hear "I don't trust AI companies with my data."?

What's this sub for?

- Questions about selling or licensing business data, from owners, operators, finance people, whoever

- Sharing what you've been offered, ads you've seen, deals that went sideways (anonymize stuff obviously)

- Discussion of the buyers, the labs, the contracts, the privacy side

Ground rules

1. Be useful. No low-effort spam or "DM me for a deal" posts.
2. Disclose if you're a buyer, broker, or vendor. I'll go first: I'm a broker/buyer.
3. Don't post anyone's actual data or identifying details.

If you want the long version, I wrote it up here: sanctusdatalabs.com/sell-business-data

Otherwise drop a question below. What's the one thing you'd want to know before selling your company's data?

For my Data Sciences course, I need to extract and compare data from two datasets, and I figured a fairly timely topic would be water consumption of AI data centers vs nuclear power generation. However, I'm having some trouble finding comparable datasets. For example, one dataset I found for data centers doesn't measure water consumption itself but rather Water Use Efficiency (WUE) (which, if I find a dataset that measures electricity produced along with withdrawn and consumed water for nuclear power I could do the math myself to figure out the WUE, but I would really rather have the actual numbers behind the WUE). It doesn't help that this is my first time really searching for data sets so I don't know how to word my queries or what sites I should be focusing on. I'm finding quite a few scholarly articles, but not a lot of the actual datasets that they used.

So far I have this dataset that tells WUE for data centers at large (not necessarily AI data centers) and this dataset (I plan on using the 2019 data) that tells water consumption of power generating plants (not just nuclear but I can pretty easily filter out what I don't need). I'm not super thrilled with either one, but I'm new to finding datasets so if you think this is the best I'm going to get let me know.

Any help would be appreciated! If you need more information or have any questions be sure to comment below and I'll be sure to get back to you. Thank you!

Hey,

I've been working on a set of AI literacy games to raise awareness for non technical people. I think they need to learn when to use AI or not, how to read numbers on a chart, the different concepts, etc.

They are free, would be happy to receive any feedback :)

https://maketools.ai/

I'm also thinking about other ways to raise awareness, like fun conferences, AI days, small workshops with characters to play. What have you seen working?

Disclosure: I maintain the curated Kaggle version linked below (minkum07). The original NFHS-3/4 survey data belongs to India's Ministry of Health and Family Welfare and IIPS, via OGD India.

A useful exercise with this dataset is to compare NFHS-4 urban and rural percentages while keeping the source's suppression and small-sample flags visible. The release has 114 indicators across 36 survey-era state/UT labels, with 16,416 tidy-long records, CSV/Parquet tables, definitions and example notebooks.

Suggested workflow:

1. Pick one percentage indicator and read its definition and denominator before joining tables.
2. Select NFHS-4 urban and rural rows, join on state and indicator, and check that the join keys are unique.
3. Calculate urban minus rural in percentage points only when both values are available. Keep missing values missing and carry the source flags into the output.
4. Make a sorted dot plot with clear missing-value and small-sample annotations. Treat it as a descriptive comparison, since confidence intervals are not included.

Don't sum total, urban and rural percentages: those populations overlap. Between-survey changes also need comparable definitions and boundaries; the package withholds Andhra Pradesh/Telangana changes where boundaries differ. These are historical aggregate data from 2005–06 and 2015–16, not current health coverage or individual records.

Original government source:

https://www.data.gov.in/resource/all-india-level-and-state-wise-key-indicators-nfhs-3-and-nfhs-4

Tables, documentation and notebooks:

https://www.kaggle.com/datasets/minkum07/india-nfhs-34-health-state-and-rural-urban-gaps

Government data and derivatives retain GODL-India; authored code is MIT. Full attribution is on the dataset page. Feedback on the join checks or how best to display the sample flags would be welcome.

I maintain Chrome UX Report Dumps, a collection of monthly Chrome UX Report website lists grouped by rank and published as compressed downloads. It’s meant to make the data easier to use without exporting it from BigQuery.

The latest update adds the August 2026 dataset: 18,294,881 entries across 10 files, totaling 94.7 MiB compressed. The repository now contains 1,064,254,496 entries across 67 monthly datasets, totaling 5.32 GiB compressed. Those are counts across monthly dumps, not a count of unique websites.

If you use website lists for research or security work, I’d welcome feedback. What would make these dumps more useful, and what features or formats would you like to see?

Hey guys, I’m looking for a dataset that has Indian names along with context in English. Something like Call Rahul tonight, Let me inform Ajay about the party, Anaya is yet to arrive at the venue etc.

It’s for a project of mine. Any dataset recommendations?
I couldn’t find anything like this on the internet

1- Quantitative -> it consists of numerical variables.It have to types - Discrete and Continuous.

2- Qualitative -> it consists of categorical data.It also have two type - Nominal and ordinal.

AI agents can query data incredibly fast, but the harder problem is knowing what that data actually means. I explore why metadata, lineage, semantic layers and governance are becoming runtime infrastructure for AI-native data platforms.

I wonder how different businesses think about using for data analytical insights, customization and predictions. When is a good time to build a data functionality. Do you have a dedicated data team in your business? What’re some reasons to decide having one or not?

Given there has been a ton of discussion around Jev for the past week, I wanted to do more research around what might be good use cases for data with that model, and I came up with the following. After chatting with Claude and ChatGPT about it, I've tried out the triage use case, and it works pretty interestingly for routing a problem to the right team and determining the severity. Hopefully you all find this helpful!