fleetmanager: backend apis and data workflowsfleetmanager: backend apis and data workflows

FleetManager was the client project I worked on during my 2025 data engineering internship at CarbonSustain. The client operates vehicle fleets and needed its trip history, utility-service territories and carbon data turned into maps and APIs it could actually use.

• Role: Data engineering intern. I built and debugged the backend APIs and data workflows alongside a frontend developer. My manager handled client communication and helped set priorities.
• Stack: Python, Django (REST APIs), PostgreSQL, AWS S3, GeoJSON/CSV processing, Postman.
• Result: Trip uploads, utility boundary lookups, carbon intensity queries and trip-profile modeling delivered as working APIs and maps. I demoed them to the client every week and directly to CarbonSustain's CEO and CTO.
• When: May–August 2025.

1. The problem

The client's data came in three unrelated shapes:

• Trip files: CSVs of historical trips, across multiple fleets.
• Utility-service data: large GeoJSON and CSV files describing which utility serves which territory.
• Carbon intensity data: the emissions context for the electricity those vehicles used.

These were files and one-off analyses. The client needed APIs another application could call, and maps with reliable data behind them.

2. Approach

TRIPS                                TERRITORIES + CARBON
trip CSVs, many fleets               utility GeoJSON/CSV · carbon data
   │                                    │
   v                                    v
Django upload API                    normalize into PostgreSQL
   ├──> S3: original + cleaned CSV      ├──> territory maps
   ├──> PostgreSQL: trip records        ├──> boundary lookups
   └──> vector search index             └──> carbon intensity queries
   │                                    │
   v                                    │
trip-profile models                     │
   │                                    │
   └────────────────┬───────────────────┘
                    v
           Django REST APIs ──> frontend maps · weekly demos

• Trip uploads: a bulk upload API stores each file in S3 and loads structured records into PostgreSQL. The upload path also feeds a vector search index.
• Utility territories: I turned the GeoJSON and CSV utility files into normalized PostgreSQL tables. Each utility gets one consistent identity across files that named it differently, and every service-area attribute stays attached to the right territory.
• Lookups vs. carbon: boundary lookups answer "which utility serves this area?", and carbon queries answer "how dirty is this electricity?". I kept them separate, because a utility's territory doesn't by itself determine its carbon intensity.
• Trip profiles: predictive models of each vehicle's usage, built from its trip history and served through the same APIs.

3. What broke: a parser built for the first file

Symptom. My trip ingestion worked on the first datasets, then started failing as more client CSVs arrived. Headers differed between datasets, some rows had more or fewer fields than others, and values were missing.

Cause. As more datasets came in, I realized my first parser had effectively hardcoded the column names from the first example files. It was built for those files, not for the client's actual data.

BEFORE: headers hardcoded from the first files

first files ──> parser expects their exact headers ──> PostgreSQL  ✓
next files  ──> other headers · uneven rows · blanks ──> fails     ✗

Fix. I redesigned ingestion around a standardization step. Each upload is normalized into one expected shape (column names, row structure, missing values) before any record is written to PostgreSQL. S3 keeps both the original CSV and the cleaned version, so any record can be traced back to exactly what the client sent and reprocessed if the rules change.

AFTER: standardize first, then insert

any client CSV
   │
   ├──> S3: original, exactly as sent
   v
standardize: header names · row shape · missing values
   │
   ├──> S3: cleaned copy
   v
PostgreSQL: structured trip records

Checking it. I tested the endpoints with Postman and showed ingestion working on the new datasets in the weekly client demos.

What's still slow. I didn't benchmark it, so there are no timings, but this is how it behaved as the input grew:

input                                 behavior
────────────────────────────────────  ─────────────────────────────
thousands of rows, multiple fleets    works
large multi-fleet requests            slow
  + processing in chunks              faster, still slower than wanted
gigabyte-scale files                  can break the system

How the work moved. Requirements changed week to week, so the project ran in a loop:

prioritize (manager + me) ──> build data + APIs ──> wire into the frontend
        ^                                                   │
        │                                                   v
new metrics or priorities <──────────────────────── weekly client demo

Keeping ingestion and storage separate from the analysis on top kept the foundation stable while priorities changed.

4. Results

• Delivered: the bulk trip upload API (S3 + PostgreSQL), utility territory tables, boundary lookups, carbon intensity queries and trip-profile modeling, all exposed as Django REST APIs.
• Ingestion handles inconsistent headers, uneven rows and missing values, instead of only the files it was first written for.
• Demos: weekly to the client, and directly to CarbonSustain's CEO and CTO.
• Handoff: by the end of my internship in August 2025, the backend functionality in my scope was in place, and the remaining features were handed off.

I didn't measure throughput, latency, model accuracy, or cost or emissions impact.

5. Next steps

1. Fix large uploads: stream files from S3 in fixed-size chunks rather than loading them whole, and benchmark the multi-fleet case before and after.
2. Add regression tests for the parser: a folder of real-world-shaped CSVs (renamed headers, uneven rows, blanks) that every parser change must pass.
3. Evaluate the trip model: compare predictions made from earlier data with the trips that actually happened later.
4. Record carbon provenance: the source, units, region and time period behind every carbon intensity value.

What I learned

My first parser fit the data I had, not the data the client would send. Next time I'd get several real samples before writing ingestion code, keep the raw input next to the cleaned output from day one, and treat "the next file looks different" as normal. The other lesson was about scope: requirements changed every week, so I'd agree on acceptance criteria up front and write down every new request before it reaches the build.