"Building a Data Pipeline: Apache Airflow and PostgreSQL Integration on WSL"

Install WSL and Ubuntu

Open PowerShell as Administrator:

Press Win + X and select Windows PowerShell (Admin).

Install WSL:

Run the following command:

powershell

wsl --install

This command installs WSL, the latest Ubuntu distribution, and sets WSL 2 as your default version.

After installation, restart your computer.

Open Ubuntu from the Start menu and complete the initial setup by creating a user and password.

🐘 Step 2: Install PostgreSQL on Ubuntu (WSL)

Update Package Lists:

sudo apt update

Install PostgreSQL:

sudo apt install postgresql postgresql-contrib

Start PostgreSQL Service:

sudo service postgresql start

Access PostgreSQL:

sudo -u postgres psql

Create Airflow Database and User:

CREATE DATABASE airflow;

CREATE USER airflow WITH PASSWORD 'airflow';

GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;

Exit PostgreSQL:

\q

🐍 Step 3: Set Up Python Virtual Environment

Install Python 3 and venv:

sudo apt install python3 python3-venv python3-pip

Create Virtual Environment:

python3 -m venv airflow_env

Activate Virtual Environment:

source airflow_env/bin/activate

pip install --upgrade pip

📦 Step 4: Install Apache Airflow

Install Apache Airflow with PostgreSQL Support:

pip install apache-airflow[postgres]

Set Airflow Home Directory:

export AIRFLOW_HOME=~/airflow

Initialize Airflow Database:

airflow db init

Create Airflow Admin User:

airflow users create \

--username admin \

--password admin \

--firstname Admin \

--lastname User \

--role Admin \

--email [email protected]

Start Airflow Web Server:

airflow webserver --port 8080

Start Airflow Scheduler:

airflow scheduler

🔗 Step 5: Connect to PostgreSQL Using DBeaver

Open DBeaver and click on the New Connection button.

Select PostgreSQL from the list of database types.

Configure Connection Settings:

Host: localhost

Port: 5432

Database: airflow

Username: airflow

Password: airflow

Test Connection to ensure it's successful.

Finish to save the connection.

🌐 Step 6: Access Airflow Web Interface

Open your web browser and navigate to:

http://localhost:8080

Log in using the credentials:

Username: admin

Password: admin