Sustainability index · Data & Automation · 2022
80k companies, one lookup tool
Context
A third-party data provider had the company records we needed, but no official API. Access was gated behind a web interface that analysts had to click through manually. The whole update cycle slowed down because the bottleneck was human navigation.
Trigger
Trigger
An analyst spent three days re-checking records that had already been gathered the previous month.
The build
- 01
Buy access, then watch the network tab
We paid for a seat on the platform. The real work was opening the browser dev tools, recording the requests, and reconstructing the undocumented endpoints and auth flow.
- 02
Automate with Python
A Python script replayed those calls with proper session handling, pagination and retries. It ran headlessly and logged every request so failures were traceable.
- 03
Structure and store
Raw responses were normalized and loaded into PostgreSQL with source timestamps, so the index was always built from the latest data rather than a stale export.
- 04
Expose to the team
Analysts could query the structured dataset through a simple lookup interface instead of opening the original tool at all.
Stack
- Python
- PostgreSQL
- Requests
- Reverse-engineering
Result
DAYS
of manual clicking reduced to a script that runs in minutes
Honestly
What I'd do differently
The data itself is never stale — it comes straight from the provider's live API, so whatever the script pulls is current. The unfinished part is the trigger: I never solved automatic renewal of the auth token, so the run still has to be kicked off by hand with a fresh session. It saves days of clicking, but it is not yet a scheduled job. Capturing the login flow properly, instead of copying a token, is the piece I would finish next.