Topics
The areas this blog keeps coming back to, read from the posts themselves.
Drag, scroll or use ← → · click the front card to open it
Writing the code itself: Python and pandas, JavaScript, web frameworks, testing, Git and AI coding tools.
Moving data in production: AWS Glue, Lambda and Step Functions, Azure Data Factory, Airflow, batch jobs, scheduling and data quality checks — what broke and how it got fixed.
The ground everything runs on: AWS accounts and IAM, networking and SSH, Linux and the shell, Docker, Terraform, CI/CD.
Computer science foundations, software architecture and the career side of the job.
SQL from the engine's point of view, PostgreSQL internals, dbt and Snowflake, dimensional modeling, OLTP vs OLAP.
Design and theory behind data systems: GFS, MapReduce, Dynamo, Spark, HDFS internals, CAP, sharding, replication.
Columnar engines and the BI layer on top: ClickHouse, DuckDB, PostHog, Power BI, Streamlit, Looker.
File and table formats, and how a data lake is laid out: Parquet, Iceberg, Delta, Hive, Trino, Databricks, partitioning, small files.
Pulling data from third parties: OAuth and tokens, rate limits, retries, webhooks, scraping, Postman, SaaS connectors.
Claims and value-based care data with Tuva and dbt: CMS, CCLF, HCC, risk adjustment and risk contracts.
Markets from the data side: market data APIs, futures and options, crypto and DeFi, systematic trading.
Field and crop data: agronomic analytics, sensors and imagery.
Map
What sits next to each blog topic in the data engineering stack. Filled nodes already have posts here; hollow ones are still unwritten.
Not written yet
- Change data capture — Data pipelines · Data modeling · Storage
- Kafka & Kinesis — Data pipelines · Distributed systems
- Flink — Distributed systems · Analytics & BI
- Redshift — Data pipelines · Analytics & BI
- Catalogs & governance — Storage · Data modeling
- Kubernetes — Distributed systems · Cloud & infra
- Indexes & query plans — Data modeling · Analytics & BI
- ELT connectors — API integration · Data pipelines
- Druid & Pinot — Analytics & BI · Distributed systems
- Hudi — Storage
- Consensus (Raft, Paxos) — Distributed systems
The US healthcare domain, laid out like the healthcare and markets roadmaps. Filled nodes already have posts here; hollow ones are still unwritten.
Not written yet
- Medicare Advantage & PMPM — Money flow · Risk adjustment
- CMS rate notices — Money flow · Risk adjustment
- PBMs & pharmacy — Money flow · Markets
- ICD-10, CPT & HCPCS — Code sets · Risk adjustment
- MS-DRG — Code sets · Claims modeling
- NDC & RxNorm — Code sets · Markets
- NPI & provider identity — Code sets · Data formats
- MMR & MOR files — Data formats · Risk adjustment
- Attribution — Claims modeling · Quality & value
- RADV audits — Risk adjustment · Compliance
- Utilization & readmissions — Quality & value · Analytics & ML
- Claims lag & IBNR — Quality & value · Markets
- Cost-effectiveness & QALYs — Markets · Quality & value
- Medicaid — Money flow
- X12 EDI (837, 835) — Data formats
- HEDIS & Star Ratings — Quality & value
- LLMs over the warehouse — Analytics & ML
- Hospitals & post-acute — Markets
- Pharma & devices — Markets
Markets, options and systematic trading, laid out like the finance roadmap; PCA and SVD come from the linear algebra one. Filled nodes already have posts here; hollow ones are still unwritten.
Not written yet
- Returns, volatility & drawdown — Fundamentals · Systematic trading
- Portfolio theory & CAPM — Fundamentals · Systematic trading
- OHLCV & corporate actions — Market data · Systematic trading
- Survivorship & look-ahead bias — Market data · Systematic trading
- Option chains & IV surfaces — Market data · Options & greeks
- GEX & expiration dynamics — Options & greeks · Markets & instruments
- Backtesting — Systematic trading · Market data
- Monte Carlo — Systematic trading · Options & greeks
- PCA & SVD — Systematic trading · Fundamentals
- NPV, IRR & compounding — Fundamentals
- DCF & multiples — Fundamentals
- Taxes on investments — Fundamentals
- Bonds & yield curve — Markets & instruments
- Futures & FX — Markets & instruments
- CEDEARs, MEP & caución — Markets & instruments
- Black-Scholes & binomial — Options & greeks
- Greeks & delta hedging — Options & greeks
- Spreads, straddles & condors — Options & greeks
- Momentum & mean reversion — Systematic trading
- Trading plan & journal — Systematic trading
- Stablecoins — Crypto & DeFi
Sensors to dashboards, laid out like the IoT roadmap, with the agro and mining ones as verticals. Filled nodes already have posts here; hollow ones are still unwritten.
Not written yet
- PLCs, Modbus & OPC UA — Devices & sensors · Mining
- Microcontrollers & SBCs — Devices & sensors · Edge computing
- MQTT & Sparkplug — Connectivity · Ingestion & streaming
- LoRaWAN & NB-IoT — Connectivity · Agro
- Satellite IoT — Connectivity · Mining
- Protobuf & payload formats — Connectivity · Ingestion & streaming
- Store-and-forward — Edge computing · Connectivity
- Node-RED & Telegraf — Edge computing · Ingestion & streaming
- k3s at the edge — Edge computing · Security & ops
- TinyML — Edge computing · Devices & sensors
- OTA & fleet management — Edge computing · Security & ops
- AWS IoT Core & Azure IoT Hub — Ingestion & streaming · Security & ops
- MQTT into Kafka — Ingestion & streaming · Time series
- Late & out-of-order events — Ingestion & streaming · Time series
- Anomaly detection — Time series · Mining
- Predictive maintenance — Time series · Mining
- Grafana & alerting — Time series · Security & ops
- OT/IT segmentation — Security & ops · Mining
- Digital twins — Security & ops · Mining
- Weather stations & soil probes — Agro · Devices & sensors
- ISOBUS & connected machinery — Agro · Connectivity
- Geospatial Python — Agro · Mining
- Drones & orthomosaics — Agro · Devices & sensors
- Yield estimation — Agro · Time series
- Fleet telemetry & dispatch — Mining · Ingestion & streaming
- SCADA & historians — Mining · Devices & sensors
- Mosquitto & EMQX — Ingestion & streaming
- TimescaleDB & InfluxDB — Time series
- ClickHouse for sensor data — Time series
- Downsampling & rollups — Time series
- Sentinel-2 & NDVI — Agro
- Water & tailings monitoring — Mining
- Drillholes & block models — Mining
From the pilot course outward: the physics of flight, weather and navigation, the aircraft, the rules, drones and flight data. Filled nodes already have posts here; hollow ones are still unwritten.
Not written yet
- Flight logs — Flight data · Drones & UAV · Pilot training
- PPL licence — Pilot training · Rules & airspace
- Stalls & spins — Aerodynamics & physics · Pilot training
- Weight & balance — Aerodynamics & physics · Aircraft systems
- METAR & TAF — Meteorology · Flight data
- Instruments & GPS — Navigation · Aircraft systems
- Airspace & radio — Rules & airspace · Navigation
- Drone regulation — Drones & UAV · Rules & airspace
- Flight controllers — Drones & UAV · Aircraft systems
- Drone mapping — Drones & UAV · Flight data
- ADS-B tracking — Flight data · Navigation
- Flight simulators — Pilot training
- Lift & drag — Aerodynamics & physics
- Clouds, fronts & icing — Meteorology
- VFR navigation & charts — Navigation
- Piston engines — Aircraft systems
- ANAC & RAAC — Rules & airspace