What it is
Data analysis is the job of turning a pile of numbers into an answer. Which of my books sold best last quarter? Did the newsletter subject lines with a question mark get more opens? Which products do customers buy together? The data might be a KDP sales report, a newsletter platform's export, a shop's order list, or a spreadsheet you've been filling in by hand for years. The analysis is everything between "here's the file" and "here's what it means": cleaning it up, calculating, charting, and explaining.
AI chat tools have become surprisingly good at this, and the reason is worth understanding, because it decides whether you can trust the result.
A language model on its own is a text predictor. It reads words and writes the words most likely to come next. That makes it good at explaining things and bad at arithmetic, because adding up 400 order totals is not a language task. If you paste a table into a plain chat and ask "what's the average?", the model may produce a number that looks right without having done any calculation at all. It's pattern-matching what an average of numbers like these usually looks like. Sometimes that guess is correct. Often it's close. Occasionally it's wildly off, and it's stated with exactly the same confidence either way. That's a form of hallucination, and it's the single most important thing to know about AI and numbers.
The better approach, which the major chat tools now use, is different. When you upload a file and ask a question, the AI writes a small program, usually in the Python programming language, and runs it in a sandbox: a sealed-off computer on the provider's side that can read your file, do the sums, draw the chart, and hand back the result, but can't reach the rest of the internet or your own machine. The model is doing what it's good at (understanding your question and writing code), and an ordinary computer is doing what it's good at (arithmetic that is exactly right every time). This is a kind of tool use: the AI reaches for a calculator instead of guessing.
The difference matters for you as a user because the two modes can look identical in the chat window. Both end with a confident paragraph and some numbers. The only reliable way to tell which one happened is to look for the code. Most tools show a small "analysis", "code", or "working" section you can expand. If it's there, the numbers came from a calculation. If there's no code, treat every figure as a guess until you've checked it.
Even when the code runs perfectly, the analysis can still be wrong, because the code does what the AI decided to do, and the AI can decide badly. It can read a column of prices as text, quietly drop rows it couldn't understand, or answer a slightly different question from the one you asked. So data analysis with AI has two layers: the calculation, which a computer does exactly, and the choices about what to calculate, which a model makes and a person should check.
Why it matters
It puts real analysis within reach of people who never learned it. A self-publisher with fifteen titles, a shop owner with three years of orders, and a newsletter writer with two hundred issues all sit on data that could answer useful questions. Until recently, getting those answers meant learning spreadsheet formulas, pivot tables, or a programming language, or paying someone who had. Now you can ask in plain English, and the AI does the technical part. That's a genuine change in who gets to make decisions from evidence rather than hunches.
It's fast at the boring parts. Much of analysis is cleaning: fixing dates written three different ways, merging two exports that label the same book differently, removing duplicate orders, splitting a "Name" column into first and last. These jobs are tedious for people and quick for code. An AI that can clean a messy export in a minute frees you for the part that needs your judgement.
It explains as it goes. A good analyst doesn't just hand you a chart; they tell you what it shows and what it doesn't. AI tools will do this on request, in whatever level of detail you want, and you can ask follow-up questions like "why did you leave out March?" in a way you never could with a formula.
Its mistakes are hard to see. This is the flip side, and it's why this element sits at "Intermediate". A wrong answer from a chatbot about history might sound odd. A wrong number from a chatbot about your sales looks exactly like a right one. Spreadsheets already had this problem long before AI, as the examples below show. AI makes analysis faster and easier, which means it also makes mistakes faster and easier, and the habit of checking becomes more important, not less.
Your data may be sensitive. Order lists contain customers' names and addresses. Newsletter exports contain email addresses. When you upload a file to a chat tool, it leaves your computer and sits on someone else's servers, subject to their terms. Whether that's acceptable depends on the data, the tool, your plan, and the law where you and your customers live. More on this under privacy, and in the steps below.
Real-world examples
ChatGPT's Code Interpreter becomes Advanced Data Analysis (2023)
OpenAI first showed a "code interpreter" in March 2023, as one of two plugins it ran itself alongside a web browser. On 6 July 2023 it began rolling the feature out to all paying ChatGPT Plus subscribers. The idea was the one described above: give the model a working Python environment in a sandbox, cut off from the internet, with some temporary storage for uploaded files, so it could calculate instead of guess. Users could upload a spreadsheet and ask for charts, averages, and cleaned-up versions to download.
When OpenAI launched ChatGPT Enterprise on 28 August 2023, it described the feature as "advanced data analysis", formerly known as Code Interpreter, and the new name stuck for a while. The name change says something: the company had noticed that most people used the code-running feature not to write software but to analyse data. Claude, Gemini, and Microsoft Copilot now offer similar abilities. Names, file limits, and which plans include them change often, so check your own tool's current terms.
Microsoft took a related route inside the spreadsheet itself. On 22 August 2023 it announced a public preview of Python in Excel, built with the data-science company Anaconda, and the feature became generally available in September 2024. Microsoft's own help pages note that the Python calculations run in the Microsoft Cloud, not on your computer. It's a useful reminder that "in my spreadsheet" doesn't always mean "on my machine".
The Reinhart-Rogoff spreadsheet error (2013)
Long before chatbots, a spreadsheet showed how a small analysis mistake can travel a long way. In 2010, the Harvard economists Carmen Reinhart and Kenneth Rogoff published a paper that was widely read as showing that countries whose government debt passed 90 percent of their economic output tended to see growth fall, with an average of minus 0.1 percent. The figure was cited in arguments for cutting government spending in several countries.
In 2013, Thomas Herndon, a graduate student at the University of Massachusetts Amherst, tried to reproduce the result for a class assignment and couldn't. With his professors Michael Ash and Robert Pollin, he obtained the original spreadsheet and found three problems. One was simple: an averaging formula didn't cover the whole range of rows, so five of the twenty countries (Australia, Austria, Belgium, Canada, and Denmark) were left out of a key calculation. The other two were choices about which data to exclude and how to weight the averages. Their April 2013 critique reported that, once these problems were corrected, average growth for the high-debt countries came out at about 2.2 percent, not minus 0.1.
The lesson for anyone using AI is not that spreadsheets are dangerous. It's that the error was invisible in the result. The output looked like a number, because it was a number. It took someone rebuilding the calculation from scratch to find it. That's exactly what "ask to see the code and check one figure by hand" is for.
England's lost Covid test results (October 2020)
In early October 2020, Public Health England disclosed that 15,841 positive Covid-19 test results, collected between 25 September and 2 October, had been left out of the daily figures. The cause, as reported by the BBC and others, was a spreadsheet file format. Test results arrived as plain text files and were pulled into Excel templates saved in the old .xls format, which has a limit of 65,536 rows. Because each test produced several rows, each template could hold only about 1,400 cases. When a file was full, extra results were simply not included. Nothing crashed and no warning appeared. The rows just weren't there.
Those cases were passed to contact tracers late, which meant many close contacts weren't warned promptly. The fix was straightforward once the problem was found, and the lesson is general: the most dangerous data errors are silent ones. When an AI tool loads your file, the equivalent risk is rows that it skipped because they had a stray character, a merged cell, or a date it couldn't read. That's why the first question to ask of any analysis is "how many rows did you load?", and the second is whether that matches what you know is in the file.
How accurate are AI data analysts? The research (2024)
Researchers have started measuring this properly. In October 2024, a team introduced DA-Code, a benchmark of data-science tasks built on real, messy datasets, in which an AI agent had to explore the data, write code, and produce an answer. Even with the best models available at the time, their agent reached only 30.5 percent accuracy. In August 2024, another group released BLADE, which compared AI agents with expert analysts on open-ended research questions. They found that models working only from their general knowledge were often limited to basic analyses, and that giving them direct access to the data helped, but their analytical choices still fell short of the experts'.
These are hard, research-grade tasks, much harder than "which book sold best in June", and models have improved since. The point for you isn't the exact score. It's that even the builders of these systems measure them getting a large share of real analysis tasks wrong, and that the failures cluster in the decisions (which data, which method, which comparison) rather than the arithmetic. Treat any single AI analysis as a capable first draft from a quick but inexperienced assistant. See benchmarks for how these tests work.
An everyday scenario: the self-publisher's royalty report
This is an illustrative scenario, not a documented case. A romance author downloads a year of royalty reports and uploads the file to a chatbot, asking which of her series earns the most per book. The AI produces a tidy chart showing her newest series well ahead. Before celebrating, she asks it to show its code and tell her how many rows it used. The code reveals that it grouped books by a series name it had inferred from the titles, and that one book with a different title pattern was left out of its series entirely. It also combined royalties from different marketplaces without converting the currencies. She asks it to fix both, then checks one figure, the total for her first series in March, against the report in a spreadsheet by hand. It matches. Now she trusts the chart, and the ranking has changed.
Try it yourself
You need an AI chatbot that can read uploaded files and run analysis. At the time of writing, ChatGPT, Claude, Gemini, and Microsoft Copilot can all do this, but free plans limit uploads and analysis, and features and terms change often, so check yours first. You also need a spreadsheet program (Excel, Google Sheets, LibreOffice, or Numbers). Allow about forty-five minutes. This exercise uses made-up data, so nothing private leaves your computer.
- Make a practice file. Ask the chatbot to create realistic fake data with some deliberate problems built in, so you can see whether the analysis catches them. Paste this:
Create a CSV file of 120 made-up orders for a small online shop that
sells printed books and ebooks. Columns: order_id, date, product,
format (Print or Ebook), quantity, price, country.
Use dates from January to June of one year.
Include these deliberate problems, and list them for me separately
at the end: two duplicate orders, three prices typed with a currency
symbol, one date written in a different format, and one row with a
missing quantity. Give me the file to download.
- Save the file and the answer key. Download the CSV and keep the list of deliberate problems somewhere you won't look at again until step 6. Open the file in your spreadsheet program once, just to see what it looks like.
- Start a fresh chat and upload the file. A new chat matters: you want an analysis that doesn't already know where the problems are. Begin with a first look, not a question:
Before any analysis: tell me how many rows and columns you loaded,
what type of data you think each column holds (number, text, date),
and list anything odd, such as missing values, duplicates, mixed
formats, or numbers stored as text. Don't fix anything yet.
- Ask three real questions. Ask them one at a time, and read each answer before the next:
1. What were total sales in money (quantity times price) for each
month? Show a simple table and a bar chart.
2. Which product earned the most, and what share of total sales
did it make up?
3. Do print or ebook orders have a higher average order value?
Tell me how many orders are in each group, and whether the
difference is big enough to mean anything with this few orders.
- Ask to see the working. After the answers, paste:
Show me the exact code or steps you used for question 1, with a
short plain-English comment on each line. Tell me how many rows
were included in the calculation, and list any rows you dropped,
changed, or couldn't read, with the reason for each.
If the tool shows no code at all, ask directly whether it ran a calculation or estimated the figures. If it estimated, ask it to calculate.
- Compare with the answer key. Now look at the list of deliberate problems from step 2. Did the first look in step 3 find all of them? Did the code in step 5 handle them, for example by removing duplicates and stripping the currency symbols, or did it quietly drop those rows? Missing any of these is common, and it's exactly what you're practising noticing.
- Check one number by hand. Pick one figure, such as total sales for March. In your spreadsheet, filter the date column to March, add a column that multiplies quantity by price, and add it up. Fix the currency-symbol prices by hand as you go. Compare with the AI's figure. If they match, you've earned some trust in the rest. If they don't, tell the AI your figure and ask it to explain the difference:
I calculated March sales by hand and got [your figure]. You reported
[its figure]. Find the reason for the difference, row by row if needed,
and tell me which of us is right.
- Try it on your own data, carefully. When you're ready to use a real export, first make a copy and delete anything personal: names, email addresses, postal addresses, phone numbers. Replace them with a plain ID if you need to tell customers apart. Check your chatbot's settings for whether uploads may be used to train future models, and whether your plan lets you switch that off. For anything covered by a client agreement or data protection law, check before uploading, not after.
Common mistakes and misunderstandings
Trusting numbers that weren't calculated. If you paste a table into a chat and get figures back with no code or visible working, the model may have estimated them. Language models are poor at exact arithmetic in plain text. Always look for the code, or ask whether a calculation was run.
Not checking what was loaded. Silent row loss is the classic spreadsheet disaster, as England's Covid figures showed. Always ask how many rows the tool loaded and compare with what you know is in the file. A difference of even one row needs an explanation.
Wrong column types. A price column containing "$12.99" and "12.99" may be read as text, and text doesn't add up. Dates like 03/04 might be March or April depending on the country. Ask the AI to list the type it assigned to each column before any analysis, and correct it where it's wrong.
Letting the AI clean without telling you. Removing duplicates, filling blanks, and dropping outliers are all reasonable steps, and all of them change the answer. Ask the AI to list every cleaning step it took, so the choices are yours.
Mistaking correlation for cause. If readers who opened your Tuesday newsletters also bought more books, it may be because Tuesday works, or because your most loyal readers happen to open everything. An AI will happily describe a pattern as if it explains something. Ask "what else could explain this?" every time.
Confident charts from tiny samples. A chart showing that ebooks outsell print by 40 percent looks authoritative whether it's based on 12 orders or 12,000. With small numbers, a few unusual orders swing everything. Ask how many data points sit behind each bar, and be wary of any conclusion drawn from a handful.
Answering a different question. "Which book sells best?" could mean most copies, most money, or most profit. The AI will pick one, often without saying which. Be precise in the question, and check which measure it actually used.
Uploading personal data without thinking. Customer lists and email exports are personal data. Remove what you don't need before uploading, and read your tool's current data terms. The analysis rarely needs anyone's name.
Believing the explanation is part of the proof. A fluent paragraph explaining why sales dipped in May is a story, not a finding. The AI can explain a wrong number just as smoothly as a right one. The proof is in the code and in the figure you checked by hand.
Thinking checking is only for beginners. The Reinhart-Rogoff error was made by leading economists, and it stayed hidden until someone rebuilt the calculation. Checking one number by hand is what careful professionals do, and it keeps a human in the loop where it counts.
Related elements
- Code Generation: writing the code that does the actual calculating behind every trustworthy AI analysis.
- Tool Use: how an AI reaches for a calculator or code sandbox instead of guessing.
- Hallucination: why a number with no working behind it can be confidently wrong.
- Structured Output: turning messy text into clean tables that are ready to analyse.
- Unsupervised Learning: finding groups in your data, such as customer types, without labels.
- Datasets: why the quality of the data decides the quality of any answer drawn from it.
- Privacy: what happens to the files you upload, and how to share less.
- Human in the Loop: keeping a person checking the figures that decisions depend on.
- Benchmarks: how researchers measure AI analysis accuracy, and what those scores do and don't tell you.
- Research: the related job of finding and weighing information, where the same habits of checking apply.