Data Engineer

Every analysis is only as good as the data under it. Before anyone draws a chart, someone has to find the right files, check for blank cells and duplicate rows, and fix dates that are written three different ways. The Data Engineer does that groundwork so you do not have to.

Think of it as the prep cook in a kitchen. Nobody serves a dish until the vegetables are washed, cut and measured. The Data Engineer washes and measures your data, then sets it out in clean, labelled containers for the rest of the team. Like a building inspector, it checks the foundations before anyone builds on them.

It also keeps a clear record. Every cleaning step is written down with counts, your original files are never touched, and it tells you plainly when a file cannot be read and why. It is the first member of the Upfyn Analyst Team, and it is the one you call when an analysis breaks.

Picture a librarian who receives a new shipment of books. Each one gets a label, a shelf and a note on its condition, so anyone can find it later and trust what they find. The Data Engineer labels each file the same way, with its rows, its columns, its quality score and what it had to fix, before any analyst opens it.

• Find your data: spreadsheets, CSV and JSON files in a folder, or data pulled from your connected apps, such as Google Sheets, Google BigQuery, Supabase, Neon, Airtable and Google Drive.
• Give you a data inventory: one table listing each file's rows, columns, quality score and issues.
• Profile the data: find blank cells, duplicate rows and unusual values, and give each file a quality score as a percentage.
• Clean it into analysis-ready copies: consistent column names, real dates and numbers, duplicate rows removed and gaps handled.
• Work out how several files relate, and flag records that do not match up across them.
• Watch a data project on a schedule, and tell you when its rows, columns or quality change.
• Diagnose a failed analysis, check the data and the tools it needs, and propose the fix.
• Describe every column: its type, how many cells are blank, its smallest and largest values, and its most common entries, so you know what each field holds before you ask a question about it.

It works in three panels: the files panel, where it reads your data in place, the browser panel for web apps, and the artifact panel, where it opens reports on the data. It never changes, moves or deletes your original files. It asks before it sends anything anywhere, sets up a schedule, or runs an optional check. Anything hard to undo waits for your yes.

The Data Engineer has seven skills. You can ask for any of them in plain words, and it picks the right one.

clean turns raw files into analysis-ready copies. It fixes data types, fills small gaps, removes duplicates and standardises column names, and logs every step. Use it for "Prepare this export so we can analyse it."

connect links a data project to your apps. That covers data sources such as Shopify, databases, Sheets, Notion and Airtable, and messaging apps such as Slack, Gmail and Discord. Use it for "Pull the orders from my Shopify store into this project."

debug diagnoses a failed analysis. It checks the tools the analysis needs, validates the data files and their encoding, finds the step that broke, and proposes a fix. Use it for "Why did the analysis fail?"

pipeline-checks is a set of optional checks that run before and after an analysis: a data check first, a run-history log, and a webhook notice. Each one runs only when you approve it. Use it when you want a safety check before a big run.

profile describes a data folder: row counts, column types, missing values, duplicates, statistics and a quality score. Use it for "How clean is this data?"

simplify reviews the code an analysis produced and tidies it. It fixes bugs only with your yes, and lists the rest as suggestions. Use it when a generated script is slow or hard to follow.

watch monitors a data project on a schedule. It logs new or removed files and changes in row counts, columns and quality, and it alerts you when quality drops. Use it for "Tell me if this data changes."

Here is a common job, from a messy file to a clean one.

1. You drop a customer export into the files panel and ask: "Clean this customer list so we can analyse it."
2. The Data Engineer lists what it found: the file name, its size, the number of rows and columns, and a first quality score.
3. It profiles the file. It reports blank cells, duplicate rows and unusual values, so you know the problems before any fix.
4. It cleans a copy. It standardises the column names, reads dates and numbers as real dates and numbers, removes exact duplicates, and fills small gaps with a sensible middle value.
5. Each step goes into a cleaning log with counts, such as how many rows were removed and which columns changed. Nothing is dropped without a line in the log.
6. Your original file stays exactly as it was. The clean copy sits in its own folder.
7. It shows you the inventory table again, this time with the new quality score next to the old one.
8. It names the records it could not fix and tells you why, so you can decide what to do with them.
9. It hands the clean copy to the Data Scientist and the Data Visualizer, and tells them it is ready.
10. You get a clean file, a log you can check line by line, and a plain note of what is still unclear.

At the end, the clean copy sits in its own folder, the cleaning log shows every change with its count, and the quality score is shown before and after. The records that still need a human decision are named, so nothing is hidden. Your original files are exactly where you left them.