AI amplifies judgment. The engineer is the edge.
I lead enterprise data platforms and build AI workflows that run inside real business systems, with controls on what they may do. Day to day, that is the Databricks platform of a global logistics company.
The routine runs on its own. Architecture and release decisions stay with me.
Platform figures updated September 2026. A step is one AI reply or one tool call; a run is the stretch between two of my messages. Runs are logged and verified. Spending money, publishing and deleting stay approval-gated.
Built and running.
One platform I run at work. One site I built independently. Every number below comes from code, logs or direct measurement.
Architecture simplified and anonymised.
The data platform I run
I run the data platform for a global logistics company. It loads 650 tables from 16 systems and feeds the company's reports. I took over the contractor-built platform in 2023 and now run it end to end.
- It loads about 700 GB automatically every weekday.
- I migrated more than 50 stored procedures and functions from T-SQL to Databricks, and proved the new numbers matched the old ones before the switch.
- I rebuilt a 34-page report as 11 smaller reports, cutting refresh time from 92.9 to 4.1 minutes.
- I am the technical lead across an 11-person BI team spread over South Africa, the USA and India.
Azure Databricks · Unity Catalog · Fabric · Power BI · GitLab CI
How the platform loads 650 tables
More than 900 configuration rows, one per configured load, tell shared loaders what to load, where it goes and how much can run at once. Most new loads need a row, not a new pipeline.
- The first loader copies source data. The second links it and records where it came from. The third builds tables for reports.
- Source loads run 10 at a time. Later steps can run up to 100 at a time.
- For the busiest source, the platform reads only what changed since the previous run.
- An automated check catches code that reads from the wrong layer.
Databricks Workflows · for-each tasks · Delta Lake · JDBC · Python
From test to production
Every change to production is reviewed, tested and deployed through GitLab. Nobody pushes straight to production. Developers work on a separate test environment.
- Every production table is cloned to test each day, inside the same governed catalog, so test never has to load from the source systems.
- Test can read production data, but it cannot change it.
- Production and test have separate networks and storage. Passwords and keys stay in a vault, never in code.
- People get access through company groups. Scheduled jobs use machine accounts.
Azure Databricks · GitLab CI · Unity Catalog · Entra ID · Key Vault
I replaced an ageing Joomla site with a static site and added online booking.
- It has a homepage and one page for each of the author's 18 books.
- Fonts, scripts and images come from its own domain. Social embeds load only when clicked.
- The old email booking option stayed available until online booking was ready.
- The scripted launch could be rolled back within five minutes. I checked the live site before calling it done.
Astro · Cloudflare Pages · Pages Functions · Resend
If you run a platform like this, or need one built and kept honest, tell me what you are trying to change.
How I work
One half runs in shadow. The other returns to the light before it ships.
The way I work is the product.
- PlanFrameSet the goal and limits.
- PlanReplayThe AI says the plan back plainly.
- ImplementRunWork in shadow.
- ReviewVerifyReturn with proof.
- ReviewShipRelease the verified result.
What repeats becomes a safeguard.
What can run without me.
The longest unattended run so far was 1 019 steps. Every run is logged and its result checked.
Watch first, then trust
I watch new automations until they are reliable. Spending money, publishing and deleting always need my approval.
Bull in a china shop, then a diamond
Build the rough version fast in a separate, safe environment. Then harden, test and document it before release.
The same setup sorts email, handles support tickets and writes meeting notes. In one supervised run, it diagnosed a fault with my home internet and logged the support request.
What still needs my judgment.
If I cannot explain a change plainly, I do not ship it.
Explain the plan first
Before work starts, the AI explains the plan in plain English. I correct any misunderstanding first.
Verify the real result
A success message is not proof. I check what people will actually see or receive.
Turn mistakes into safeguards
When a mistake repeats, I turn it into a rule or an automated check. Serious risks get a safeguard immediately.
Start a conversation
Tell me what you are trying to change, what is blocking it, and the outcome you need.
Email KosieKosie Roux · Technical Lead Data Engineer · Gqeberha, South Africa · LinkedIn