Data Engineering, Databricks & DevOps Blog by Stefan Kochhttps://stefanko.ch/https://stefanko.ch/p/nexus-stack/https://stefanko.ch/p/nexus-stack/Deploy your Data Stack with Docker and Cloudflare in an automated, secure, and scalable way.Thu, 22 Jan 2026 17:12:11 GMTDevOpsDataStackHetznerOpenTofuhttps://stefanko.ch/p/zero-entry-docker/https://stefanko.ch/p/zero-entry-docker/Self-hosting is awesome – you have full control over your data, no subscription fees, and you learn a lot about system administration along the way. But it comes with real challenges: security concernTue, 06 Jan 2026 18:00:00 GMThowtoself-hostinginfrastructure-as-codehttps://stefanko.ch/p/adventureworks-simulator/https://stefanko.ch/p/adventureworks-simulator/If you've ever tried to learn Change Data Capture (CDC), streaming pipelines, or incremental data loading, you've probably hit the same frustrating wall I did: **static sample databases don't teach yoTue, 09 Dec 2025 06:00:00 GMTData EngineeringAzurehttps://stefanko.ch/p/databricks-sharepoint-integration/https://stefanko.ch/p/databricks-sharepoint-integration/In modern data workflows, it’s essential to collect data where it originates and deliver it to where it’s needed. In this blog post, I’ll show how to connect **SharePoint directly with Databricks** – Wed, 28 May 2025 07:20:17 GMThowtodatabricksintegrationSharePointhttps://stefanko.ch/p/delta-live-table-full-snapshot-source/https://stefanko.ch/p/delta-live-table-full-snapshot-source/In this article I describe how to load data from recurring full snapshots with Delta Live Tables relatively easily and elegantly into a bronze table without the amount of data exploding.Sun, 16 Mar 2025 12:31:27 GMThowtodatabricksBusiness Intelligencehttps://stefanko.ch/p/databricks-sql-warehouse-as-a-linked-server/https://stefanko.ch/p/databricks-sql-warehouse-as-a-linked-server/In this blog post, I explore how to set up Databricks SQL Warehouse as a Linked Server in MS SQL Server to seamlessly query data from your Lakehouse directly within SQL Server. Thu, 09 Jan 2025 17:15:57 GMTDatabricksSQL Serverhttps://stefanko.ch/p/effortless-databricks-asset-bundle-deployments-with-azure-devops/https://stefanko.ch/p/effortless-databricks-asset-bundle-deployments-with-azure-devops/In this step-by-step tutorial, you’ll learn how to set up an Azure DevOps pipeline for seamless deployment of Databricks asset bundles. The post walks you through the entire process—from configuring your Azure DevOps project to automating the deployment of Notebooks, Workflows, and other Databricks resources.Wed, 08 Jan 2025 19:47:57 GMTdatabricksDevOpsCI/CDhttps://stefanko.ch/p/create-dim-date-time-in-databricks/https://stefanko.ch/p/create-dim-date-time-in-databricks/In this article, you will learn how to set up date and time dimensions in Databricks to enable precise time-based analyses and reports.Sun, 10 Nov 2024 07:51:57 GMThowtodatabricksBusiness Intelligencehttps://stefanko.ch/p/trigger-and-monitor-adf-from-databricks/https://stefanko.ch/p/trigger-and-monitor-adf-from-databricks/In data engineering in the Azure Cloud, a common setup is to use Azure Data Factory to orchestrate data pipelines. If you wanted to orchestrate Databricks pipelines, you had a powerful tool at hand wiFri, 01 Nov 2024 07:11:21 GMThowtodatabricksdata factoryhttps://stefanko.ch/p/databricks-cli-tutorial-1/https://stefanko.ch/p/databricks-cli-tutorial-1/In the upcoming blog series, I will highlight different areas of the Databricks CLI with various practical examples.Tue, 29 Oct 2024 19:04:16 GMThowtodatabrickshttps://stefanko.ch/p/databricks-cli-authentication/https://stefanko.ch/p/databricks-cli-authentication/I will show you different methods to authenticate with the Databricks CLI to Azure Databricks in this quick guide.Wed, 23 Oct 2024 18:29:51 GMThowtodatabrickshttps://stefanko.ch/p/openweather/https://stefanko.ch/p/openweather/In this HowTo I will show you how to load weather data from the OpenWeather API into the Databricks Lakehouse. The data is loaded via REST API from OpenWeather and then processed in the Medallion archThu, 21 Mar 2024 06:51:57 GMThowtoDatabricksETLhttps://stefanko.ch/p/copy-schema-tables-databricks/https://stefanko.ch/p/copy-schema-tables-databricks/In this quick guide, I will show you how to automatically copy all schemas and tables from one catalog to another in Databricks.Sun, 17 Mar 2024 05:51:57 GMThowtodatabrickshttps://stefanko.ch/p/dbt-core-databricks/https://stefanko.ch/p/dbt-core-databricks/In this step-by-step guide, I will show you how to install dbt core locally in Visual Studio Code, set up the corresponding Visual Studio Code Extension and run dbt on Databricks.Sun, 10 Mar 2024 08:51:57 GMThowtodbtDatabrickshttps://stefanko.ch/p/airbyte-vm/https://stefanko.ch/p/airbyte-vm/In this quick guide, I will show you how to set up an Airbyte environment in a virtual machine in Azure.Sun, 03 Mar 2024 09:51:57 GMThowtoazureETLhttps://stefanko.ch/p/databricks_lakehouse/https://stefanko.ch/p/databricks_lakehouse/In this step-by-step guide, I describe how to create a Databricks Lakehouse environment in the Azure Cloud within 30 minutes. This guide is aimed at people who are new to Databricks and want to createSun, 18 Feb 2024 09:51:57 GMThowtodatabrickshttps://stefanko.ch/p/medium/https://stefanko.ch/p/medium/I have now decided in the old year to write on Medium as well as my personal blog.Tue, 12 Dec 2023 11:51:57 GMTgeneralblogginghttps://stefanko.ch/p/First_steps_with_the_Azure_Cloud/https://stefanko.ch/p/First_steps_with_the_Azure_Cloud/Azure is Microsoft’s cloud computing platform. It offers a comprehensive suite of cloud services that enable companies to develop, deploy and manage applications without having to have physical hardware on site. Azure enables users to use resources such as virtual machines, storage, databases, networks and much more in Microsoft’s globally distributed data centers. The Azure Cloud is one of the leading hyper-scalers on the market. In the following article, I would like to give beginners a few useful tips to make it easier to get started with the Azure portal.Mon, 11 Dec 2023 06:51:57 GMThowtoazurehttps://stefanko.ch/p/DBeaver_SQL_Warehouse/https://stefanko.ch/p/DBeaver_SQL_Warehouse/In this quick guide I will show you how to connect to a Databricks SQL Warehouse Cluster with DBeaver.Tue, 28 Nov 2023 08:51:57 GMThowtodatabricks