Danil Sidorov
RU EN DE
Telegram

A data lake for sales exports

Uploading CSV files from accounting systems with quality checks, layers from the source file to Parquet and charts

Аналитика / Обработка данных

Live interactive demo stand

An exact copy of the project interface and business logic with demo data. All actions run in real time.

Битая выгрузка не попадёт в отчёты: проверка до записи

StackPython, Django, pandas, pyarrow, Parquet, Plotly, Bootstrap

Safe perimeter (NDA): production logic on test data

What to test in the stand:
  1. Загрузите CSV-файл продаж и проверьте валидацию на дубликаты и пропуски
  2. Посмотрите 4 слоя преобразования данных от сырого файла до Parquet
  3. Постройте интерактивный график выручки и скачайте очищенную выгрузку
http://127.0.0.1:8000/admin/main/data/
Live Stand
DLDataLake
analyst · staff
Администрирование Django
Начало › Main › Неструктурированные данные

Выберите неструктурированные данные для просмотра

Время загрузкиФайлРазделительДействия

Добавить неструктурированные данные

Слои последней загрузки

Turnkey implementation

Need a similar system for your business processes?

I will develop the architecture, fault-tolerant backend, and fast interface. Protected by NDA, clean transferable code, and transparent milestones.

  1. Free timeline and budget estimate within one day
  2. Contract and invoice
  3. Warranty and post-launch support

On weekdays usually within a couple of hours

I will reply personally, no aggressive sales pitch

Task

Sales exports came from different systems in different encodings and with different separators, with repeats and empty cells. They had to land in one place without letting broken files through, and the data had to be easy to look at without Excel.

Solution

Built a web application on Django and pandas. On save the file is copied to the raw layer, read with a fallback over encodings and run through the quality checks. If a check fails, every layer of that upload is removed and the user sees the list of errors. Otherwise the data goes to ingest, table and destilled, where the preview, the charts and the export read it.

What was done

  • Built CSV uploads in the admin with a choice of separator
  • Detect the encoding automatically: UTF-8, UTF-8 with BOM or Windows-1251
  • Check the file before saving: repeated rows and gaps reject the upload
  • Lay the data out in layers: source, UTF-8 CSV, typed Parquet and deduplicated Parquet
  • Built a preview of the first rows of a dataset
  • Built a chart builder where columns are dragged onto the axes
  • Show column statistics and export the dataset back to CSV

Result

A broken export never reaches the data: the check stops it and shows what to fix. A dataset can be previewed, charted and exported back without Excel. The code is open on GitHub.

Technologies

PythonDjangopandaspyarrowParquetPlotlyBootstrap

Sources

The links confirm the numbers in this case

Tell me about your task

Telegram