I am thinking of developing a process through Claude code, where I extract all the news from DEWA website, with it's podcast, send it to a creates dashboard, to check and track all the Press release with their podcast, so the dashboard will show me, the data of each PR, how many published daily, weekly and monthly, the cost of each podcast before, and after using NotebookLM so it will keep showing me the ROI before we used to pay 20,000 aed for each single podcast, but now we are only paying the monthly subscription of NotebookLM around 20$


## plugin:context7:context7
Use this server to fetch current documentation whenever the user asks about a library, framework, SDK, API, CLI tool, or cloud service -- even well-known ones like React, Next.js, Prisma, Express, Tailwind, Django, or Spring Boot. This includes API syntax, configuration, version migration, library-specific debugging, setup instructions, and CLI tool usage. Use even when you think you know the answer -- your training data may not reflect recent changes. Prefer this over web search for library docs.

Do not use for: refactoring, writing scripts from scratch, debugging business logic, code review, or general programming concepts.


Great idea! Before I start building, let me check what DEWA MCP tools are available and explore the DEWA website structure to plan the best approach.


ToolSearch

Question: select:mcp__claude_ai_DEWA__get_dewa_services


Question: select:mcp__claude_ai_DEWA__get_dewa_services


Found 11 DEWA services.

Bill Payment

View & pay your DEWA bills quickly and securely.

EasyPay

Make different types of payments in an easy way.

Activation of Electricity/Water (Move-in)

I need to activate electricity and water for my new Premise.

Transfer of Electricity/Water (Move-to)

I need to transfer electricity and water to another premise.

Deactivation of Electricity/Water (Move-out)

I want to deactivate electricity and water and request for final bill.

Attending Technical Notifications Electricity / Water (Smart Response)

Report & track electricity/water technical incident

EV Charging

Charge your EV smartly by scanning the QR code.

Found 11 DEWA services.

Found 11 DEWA services.


ToolSearch

Question: select:WebFetch


Question: select:WebFetch


WebFetch

Prompt: Extract the structure of press releases - titles, dates, URLs, any pagination info, and how podcasts are linked if any


Request failed with status code 403

Error: Request failed with status code 403


Prompt: List all available news, media, and podcast sections with their URLs and structure


Request failed with status code 403


Prompt: List all podcast entries with titles, dates, URLs and any cost or metadata visible


The DEWA website returns 403 for direct HTTP requests — it requires a real browser session. Here's my plan for the full system:

---

## System Architecture

```
┌─────────────────────────────────────────────────────────┐
│                    DEWA PR Tracker                       │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  1. SCRAPER (Playwright)                                 │
│     └─ Headless browser → DEWA press releases page       │
│     └─ Extract: title, date, URL, linked podcast         │
│     └─ Runs on schedule (daily cron)                     │
│                                                          │
│  2. DATABASE (SQLite)                                    │
│     └─ press_releases: id, title, date, url, has_podcast │
│     └─ podcasts: id, pr_id, title, date, duration        │
│                                                          │
│  3. DASHBOARD (Python Streamlit)                         │
│     └─ PRs table with podcast status                     │
│     └─ Daily / Weekly / Monthly publication counts       │
│     └─ ROI Panel:                                        │
│          Before: 20,000 AED × N podcasts                 │
│          After:  20 USD/month (~73 AED)                  │
│          Savings: Auto-calculated                        │
│                                                          │
└─────────────────────────────────────────────────────────┘
```

**Tech Stack:**
- `playwright` — browser automation (bypasses 403)
- `sqlite3` — lightweight local DB
- `streamlit` — fast, beautiful dashboard with no frontend code
- `apscheduler` — scheduled auto-scraping
- `pandas` — data manipulation for charts

**ROI Logic:**
- Old cost: `20,000 AED × total podcasts`
- New cost: `73 AED/month × months active` (20 USD ≈ 73 AED)
- Savings & ROI % auto-calculated

---

A few quick questions before I build:

1. **Where do you want this to run?** Locally on your Mac, or deployed to a server?
2. **Do you have access to the DEWA website while logged in?** (The scraper can reuse your browser session if needed)
3. **What's your preferred language?** Python is my recommendation — are you comfortable with that?
4. **Do you already have a list of press releases/podcasts**, or should the scraper build the database from scratch by crawling the DEWA site?
5. **Should the dashboard be password-protected**, or is it for internal use on a trusted network?


- Locally on my mac
- No need it's already public for everyone --> https://www.dewa.gov.ae/en/about-us/media-publications/latest-news
- Choose the best langauge you prefer
- All the the press relaeases and the RSS podcasts are there in the link i shared
- No need to be password protected


Let me first inspect the actual page structure before writing a single line of code.


Prompt: Extract the full HTML structure of news/press release listings: article titles, dates, URLs, any podcast links, pagination structure, and any RSS feed links. Show me the exact HTML patterns used for each news item.


Prompt: Show full RSS feed XML content including all items with titles, dates, URLs, descriptions, and any podcast/audio links


Prompt: Show full RSS feed content with all items, titles, dates, links


Prompt: Extract press release listing structure, article titles, dates, URLs


Prompt: Extract podcast listing structure, titles, dates, URLs, audio file links


The site blocks all automated HTTP clients — Playwright is definitely the right call since it uses a real Chromium browser. Since I can't inspect the HTML in advance, I'll build the scraper with a **discovery mode** that saves the page HTML so we can verify selectors after the first run. Let me build the full system now.


Bash

List home directory contents


Applications
course-dashboard.html
Desktop
Documents
Downloads
Library
Movies
Music
Pictures
Projects
Public
Sites
skills-lock.json


List Projects directory


finally


mkdir -p /Users/khalidbinkalban/Projects/dewa-pr-tracker && ls /Users/khalidbinkalban/Projects/dewa-pr-tracker

Create project directory


(Bash completed with no output)


Good. Now I'll build all the files:


Write

"""
DEWA PR Tracker — Configuration
"""

# ── URLs ──────────────────────────────────────────────────────────────────────
DEWA_NEWS_URL    = "https://www.dewa.gov.ae/en/about-us/media-publications/latest-news"
DEWA_PODCAST_URL = "https://www.dewa.gov.ae/en/about-us/media-publications/podcasts"
DEWA_BASE_URL    = "https://www.dewa.gov.ae"

# ── Database ──────────────────────────────────────────────────────────────────
DB_PATH = "dewa_tracker.db"

# ── ROI Settings ─────────────────────────────────────────────────────────────
COST_PER_PODCAST_OLD_AED = 20_000   # what a single podcast used to cost
NOTEBOOKLM_MONTHLY_USD   = 20       # current NotebookLM subscription
USD_TO_AED               = 3.67    # exchange rate

# ── Scraper ───────────────────────────────────────────────────────────────────
SCRAPER_HEADLESS     = True         # set False to watch the browser during debug
SCRAPER_TIMEOUT_MS   = 30_000
SCRAPER_WAIT_AFTER_LOAD_MS = 3_000  # wait for JS to finish rendering

# ── Dashboard ─────────────────────────────────────────────────────────────────
DASHBOARD_TITLE = "DEWA Press Release & Podcast Tracker"
PAGE_SIZE       = 25                # rows per page in the PR table


File created successfully at: /Users/khalidbinkalban/Projects/dewa-pr-tracker/config.py

"""
DEWA PR Tracker — Configuration
"""

# ── URLs ──────────────────────────────────────────────────────────────────────
DEWA_NEWS_URL    = "https://www.dewa.gov.ae/en/about-us/media-publications/latest-news"
DEWA_PODCAST_URL = "https://www.dewa.gov.ae/en/about-us/media-publications/podcasts"
DEWA_BASE_URL    = "https://www.dewa.gov.ae"

# ── Database ──────────────────────────────────────────────────────────────────
DB_PATH = "dewa_tracker.db"

# ── ROI Settings ─────────────────────────────────────────────────────────────
COST_PER_PODCAST_OLD_AED = 20_000   # what a single podcast used to cost
NOTEBOOKLM_MONTHLY_USD   = 20       # current NotebookLM subscription
USD_TO_AED               = 3.67    # exchange rate

# ── Scraper ───────────────────────────────────────────────────────────────────
SCRAPER_HEADLESS     = True         # set False to watch the browser during debug
SCRAPER_TIMEOUT_MS   = 30_000
SCRAPER_WAIT_AFTER_LOAD_MS = 3_000  # wait for JS to finish rendering

# ── Dashboard ─────────────────────────────────────────────────────────────────
DASHBOARD_TITLE = "DEWA Press Release & Podcast Tracker"
PAGE_SIZE       = 25                # rows per page in the PR table


"""
DEWA PR Tracker — Database layer (SQLite)
"""
import sqlite3
from contextlib import contextmanager
from datetime import datetime
from config import DB_PATH

# ── Schema ────────────────────────────────────────────────────────────────────

SCHEMA = """
CREATE TABLE IF NOT EXISTS press_releases (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    title       TEXT    NOT NULL,
    url         TEXT    UNIQUE NOT NULL,
    date        TEXT,                    -- ISO-8601 (YYYY-MM-DD)
    category    TEXT,
    description TEXT,
    scraped_at  TEXT    NOT NULL
);

CREATE TABLE IF NOT EXISTS podcasts (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    pr_id       INTEGER REFERENCES press_releases(id),
    title       TEXT    NOT NULL,
    url         TEXT    UNIQUE NOT NULL,
    audio_url   TEXT,
    date        TEXT,
    duration    TEXT,
    scraped_at  TEXT    NOT NULL
);

CREATE TABLE IF NOT EXISTS scrape_log (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    ran_at      TEXT NOT NULL,
    prs_found   INTEGER DEFAULT 0,
    pods_found  INTEGER DEFAULT 0,
    status      TEXT,
    notes       TEXT
);
"""

@contextmanager
def get_conn():
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    try:
        yield conn
        conn.commit()
    finally:
        conn.close()

def init_db():
    with get_conn() as conn:
        conn.executescript(SCHEMA)

# ── Upsert helpers ────────────────────────────────────────────────────────────

def upsert_press_release(title: str, url: str, date: str = None,
                          category: str = None, description: str = None) -> int:
    """Insert or update a press release. Returns the row id."""
    now = datetime.utcnow().isoformat()
    with get_conn() as conn:
        cur = conn.execute(
            """
            INSERT INTO press_releases (title, url, date, category, description, scraped_at)
            VALUES (?, ?, ?, ?, ?, ?)
            ON CONFLICT(url) DO UPDATE SET
                title       = excluded.title,
                date        = excluded.date,
                category    = excluded.category,
                description = excluded.description,
                scraped_at  = excluded.scraped_at
            """,
            (title, url, date, category, description, now),
        )
        if cur.lastrowid:
            return cur.lastrowid
        row = conn.execute("SELECT id FROM press_releases WHERE url = ?", (url,)).fetchone()
        return row["id"]

def upsert_podcast(title: str, url: str, pr_id: int = None,
                   audio_url: str = None, date: str = None, duration: str = None) -> int:
    now = datetime.utcnow().isoformat()
    with get_conn() as conn:
        cur = conn.execute(
            """
            INSERT INTO podcasts (pr_id, title, url, audio_url, date, duration, scraped_at)
            VALUES (?, ?, ?, ?, ?, ?, ?)
            ON CONFLICT(url) DO UPDATE SET
                title     = excluded.title,
                audio_url = excluded.audio_url,
                date      = excluded.date,
                duration  = excluded.duration,
                scraped_at= excluded.scraped_at
            """,
            (pr_id, title, url, audio_url, date, duration, now),
        )
        if cur.lastrowid:
            return cur.lastrowid
        row = conn.execute("SELECT id FROM podcasts WHERE url = ?", (url,)).fetchone()
        return row["id"]

def log_scrape(prs_found: int, pods_found: int, status: str, notes: str = ""):
    with get_conn() as conn:
        conn.execute(
            "INSERT INTO scrape_log (ran_at, prs_found, pods_found, status, notes) VALUES (?,?,?,?,?)",
            (datetime.utcnow().isoformat(), prs_found, pods_found, status, notes),
        )

# ── Query helpers ─────────────────────────────────────────────────────────────

def get_all_press_releases():
    with get_conn() as conn:
        return conn.execute(
            """
            SELECT
                pr.id,
                pr.title,
                pr.url,
                pr.date,
                pr.category,
                pr.description,
                pr.scraped_at,
                COUNT(p.id) AS podcast_count
            FROM press_releases pr
            LEFT JOIN podcasts p ON p.pr_id = pr.id
            GROUP BY pr.id
            ORDER BY pr.date DESC, pr.id DESC
            """
        ).fetchall()

def get_all_podcasts():
    with get_conn() as conn:
        return conn.execute(
            """
            SELECT p.*, pr.title AS pr_title
            FROM podcasts p
            LEFT JOIN press_releases pr ON pr.id = p.pr_id
            ORDER BY p.date DESC, p.id DESC
            """
        ).fetchall()

def get_counts_by_day():
    with get_conn() as conn:
        return conn.execute(
            "SELECT date, COUNT(*) as count FROM press_releases WHERE date IS NOT NULL GROUP BY date ORDER BY date"
        ).fetchall()

def get_counts_by_month():
    with get_conn() as conn:
        return conn.execute(
            """
            SELECT substr(date,1,7) AS month, COUNT(*) AS count
            FROM press_releases WHERE date IS NOT NULL
            GROUP BY month ORDER BY month
            """
        ).fetchall()

def get_scrape_log():
    with get_conn() as conn:
        return conn.execute(
            "SELECT * FROM scrape_log ORDER BY ran_at DESC LIMIT 50"
        ).fetchall()

def get_stats():
    with get_conn() as conn:
        total_prs    = conn.execute("SELECT COUNT(*) FROM press_releases").fetchone()[0]
        total_pods   = conn.execute("SELECT COUNT(*) FROM podcasts").fetchone()[0]
        last_scraped = conn.execute("SELECT MAX(scraped_at) FROM press_releases").fetchone()[0]
        return {"total_prs": total_prs, "total_podcasts": total_pods, "last_scraped": last_scraped}


File created successfully at: /Users/khalidbinkalban/Projects/dewa-pr-tracker/database.py

"""
DEWA PR Tracker — Database layer (SQLite)
"""
import sqlite3
from contextlib import contextmanager
from datetime import datetime
from config import DB_PATH

# ── Schema ────────────────────────────────────────────────────────────────────

SCHEMA = """
CREATE TABLE IF NOT EXISTS press_releases (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    title       TEXT    NOT NULL,
    url         TEXT    UNIQUE NOT NULL,
    date        TEXT,                    -- ISO-8601 (YYYY-MM-DD)
    category    TEXT,
    description TEXT,
    scraped_at  TEXT    NOT NULL
);

CREATE TABLE IF NOT EXISTS podcasts (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    pr_id       INTEGER REFERENCES press_releases(id),
    title       TEXT    NOT NULL,
    url         TEXT    UNIQUE NOT NULL,
    audio_url   TEXT,
    date        TEXT,
    duration    TEXT,
    scraped_at  TEXT    NOT NULL
);

CREATE TABLE IF NOT EXISTS scrape_log (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    ran_at      TEXT NOT NULL,
    prs_found   INTEGER DEFAULT 0,
    pods_found  INTEGER DEFAULT 0,
    status      TEXT,
    notes       TEXT
);
"""

@contextmanager
def get_conn():
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    try:
        yield conn
        conn.commit()
    finally:
        conn.close()

def init_db():
    with get_conn() as conn:
        conn.executescript(SCHEMA)

# ── Upsert helpers ────────────────────────────────────────────────────────────

def upsert_press_release(title: str, url: str, date: str = None,
                          category: str = None, description: str = None) -> int:
    """Insert or update a press release. Returns the row id."""
    now = datetime.utcnow().isoformat()
    with get_conn() as conn:
        cur = conn.execute(
            """
            INSERT INTO press_releases (title, url, date, category, description, scraped_at)
            VALUES (?, ?, ?, ?, ?, ?)
            ON CONFLICT(url) DO UPDATE SET
                title       = excluded.title,
                date        = excluded.date,
                category    = excluded.category,
                description = excluded.description,
                scraped_at  = excluded.scraped_at
            """,
            (title, url, date, category, description, now),
        )
        if cur.lastrowid:
            return cur.lastrowid
        row = conn.execute("SELECT id FROM press_releases WHERE url = ?", (url,)).fetchone()
        return row["id"]

def upsert_podcast(title: str, url: str, pr_id: int = None,
                   audio_url: str = None, date: str = None, duration: str = None) -> int:
    now = datetime.utcnow().isoformat()
    with get_conn() as conn:
        cur = conn.execute(
            """
            INSERT INTO podcasts (pr_id, title, url, audio_url, date, duration, scraped_at)
            VALUES (?, ?, ?, ?, ?, ?, ?)
            ON CONFLICT(url) DO UPDATE SET
                title     = excluded.title,
                audio_url = excluded.audio_url,
                date      = excluded.date,
                duration  = excluded.duration,
                scraped_at= excluded.scraped_at
            """,
            (pr_id, title, url, audio_url, date, duration, now),
        )
        if cur.lastrowid:
            return cur.lastrowid
        row = conn.execute("SELECT id FROM podcasts WHERE url = ?", (url,)).fetchone()
        return row["id"]

def log_scrape(prs_found: int, pods_found: int, status: str, notes: str = ""):
    with get_conn() as conn:
        conn.execute(
            "INSERT INTO scrape_log (ran_at, prs_found, pods_found, status, notes) VALUES (?,?,?,?,?)",
            (datetime.utcnow().isoformat(), prs_found, pods_found, status, notes),
        )

# ── Query helpers ─────────────────────────────────────────────────────────────

def get_all_press_releases():
    with get_conn() as conn:
        return conn.execute(
            """
            SELECT
                pr.id,
                pr.title,
                pr.url,
                pr.date,
                pr.category,
                pr.description,
                pr.scraped_at,
                COUNT(p.id) AS podcast_count
            FROM press_releases pr
            LEFT JOIN podcasts p ON p.pr_id = pr.id
            GROUP BY pr.id
            ORDER BY pr.date DESC, pr.id DESC
            """
        ).fetchall()

def get_all_podcasts():
    with get_conn() as conn:
        return conn.execute(
            """
            SELECT p.*, pr.title AS pr_title
            FROM podcasts p
            LEFT JOIN press_releases pr ON pr.id = p.pr_id
            ORDER BY p.date DESC, p.id DESC
            """
        ).fetchall()

def get_counts_by_day():
    with get_conn() as conn:
        return conn.execute(
            "SELECT date, COUNT(*) as count FROM press_releases WHERE date IS NOT NULL GROUP BY date ORDER BY date"
        ).fetchall()

def get_counts_by_month():
    with get_conn() as conn:
        return conn.execute(
            """
            SELECT substr(date,1,7) AS month, COUNT(*) AS count
            FROM press_releases WHERE date IS NOT NULL
            GROUP BY month ORDER BY month
            """
        ).fetchall()

def get_scrape_log():
    with get_conn() as conn:
        return conn.execute(
            "SELECT * FROM scrape_log ORDER BY ran_at DESC LIMIT 50"
        ).fetchall()

def get_stats():
    with get_conn() as conn:
        total_prs    = conn.execute("SELECT COUNT(*) FROM press_releases").fetchone()[0]
        total_pods   = conn.execute("SELECT COUNT(*) FROM podcasts").fetchone()[0]
        last_scraped = conn.execute("SELECT MAX(scraped_at) FROM press_releases").fetchone()[0]
        return {"total_prs": total_prs, "total_podcasts": total_pods, "last_scraped": last_scraped}


"""
DEWA PR Tracker — Playwright-based scraper
Run directly:  python scraper.py
Debug mode:    python scraper.py --debug   (saves HTML + screenshot, opens browser)
"""
import argparse
import re
import sys
from datetime import datetime
from pathlib import Path

from playwright.sync_api import sync_playwright, TimeoutError as PWTimeout

import database as db
from config import (
    DEWA_NEWS_URL, DEWA_PODCAST_URL, DEWA_BASE_URL,
    SCRAPER_HEADLESS, SCRAPER_TIMEOUT_MS, SCRAPER_WAIT_AFTER_LOAD_MS,
)

# ── Selector candidates (tried in order until one matches) ───────────────────
PR_ITEM_SELECTORS = [
    "article",
    ".news-item",
    ".press-release-item",
    ".listing-item",
    ".media-item",
    "[class*='news']",
    "[class*='article']",
    "[class*='press']",
    ".container li",
    ".results-list li",
]

TITLE_SELECTORS  = ["h1", "h2", "h3", "h4", ".title", "[class*='title']", "a"]
DATE_SELECTORS   = ["time", ".date", "[class*='date']", "[datetime]", "span.label"]

PODCAST_ITEM_SELECTORS = [
    ".podcast-item",
    "article",
    ".media-item",
    "[class*='podcast']",
    ".listing-item",
]

# ── Utilities ─────────────────────────────────────────────────────────────────

def normalise_url(href: str) -> str:
    if not href:
        return ""
    href = href.strip()
    if href.startswith("http"):
        return href
    if href.startswith("/"):
        return DEWA_BASE_URL + href
    return href

def parse_date(raw: str) -> str:
    """Try to parse a messy date string into YYYY-MM-DD."""
    if not raw:
        return None
    raw = raw.strip()
    formats = [
        "%d %B %Y", "%B %d, %Y", "%d/%m/%Y",
        "%Y-%m-%d", "%d-%m-%Y", "%d %b %Y",
        "%B %Y",
    ]
    for fmt in formats:
        try:
            return datetime.strptime(raw, fmt).strftime("%Y-%m-%d")
        except ValueError:
            continue
    # Fallback: grab 4-digit year
    m = re.search(r"(\d{4})", raw)
    return m.group(1) + "-01-01" if m else None

def first_match(element, selectors):
    for sel in selectors:
        try:
            el = element.query_selector(sel)
            if el:
                return el
        except Exception:
            continue
    return None

# ── Core scraping logic ───────────────────────────────────────────────────────

def scrape_press_releases(page, debug: bool = False):
    print(f"[scraper] Loading news page → {DEWA_NEWS_URL}")
    page.goto(DEWA_NEWS_URL, timeout=SCRAPER_TIMEOUT_MS, wait_until="networkidle")
    page.wait_for_timeout(SCRAPER_WAIT_AFTER_LOAD_MS)

    if debug:
        page.screenshot(path="debug_news.png", full_page=True)
        Path("debug_news.html").write_text(page.content(), encoding="utf-8")
        print("[debug] Saved debug_news.png + debug_news.html")

    # Auto-scroll to trigger lazy-loading
    for _ in range(5):
        page.evaluate("window.scrollBy(0, window.innerHeight)")
        page.wait_for_timeout(800)

    # Try "Load more" / pagination buttons
    for _ in range(20):
        try:
            btn = page.query_selector(
                "button:has-text('Load more'), button:has-text('Show more'), "
                "a:has-text('Load more'), a:has-text('Next'), .pagination .next"
            )
            if btn and btn.is_visible():
                btn.click()
                page.wait_for_timeout(2000)
            else:
                break
        except Exception:
            break

    # Find the container with news items
    items = []
    for sel in PR_ITEM_SELECTORS:
        found = page.query_selector_all(sel)
        if len(found) >= 3:
            items = found
            print(f"[scraper] Using selector '{sel}' → {len(found)} items found")
            break

    if not items:
        print("[scraper] WARNING: No items found with known selectors. Run with --debug to inspect HTML.")
        return []

    results = []
    for item in items:
        try:
            # Title + URL
            title_el = first_match(item, TITLE_SELECTORS)
            if not title_el:
                continue
            title = title_el.inner_text().strip()
            href  = title_el.get_attribute("href") or ""
            if not href:
                link_el = item.query_selector("a")
                href = link_el.get_attribute("href") if link_el else ""
            url = normalise_url(href)

            if not title or not url:
                continue

            # Date
            date_el = first_match(item, DATE_SELECTORS)
            raw_date = ""
            if date_el:
                raw_date = date_el.get_attribute("datetime") or date_el.inner_text()
            date = parse_date(raw_date)

            # Category / description
            desc_el = item.query_selector("p, .description, .excerpt, .summary")
            description = desc_el.inner_text().strip() if desc_el else None

            cat_el = item.query_selector(".category, .tag, .label, [class*='categ']")
            category = cat_el.inner_text().strip() if cat_el else None

            results.append({
                "title": title,
                "url": url,
                "date": date,
                "category": category,
                "description": description,
            })
        except Exception as e:
            print(f"[scraper] Skipping item due to error: {e}")
            continue

    print(f"[scraper] Extracted {len(results)} press releases")
    return results

def scrape_podcasts(page, debug: bool = False):
    print(f"[scraper] Loading podcasts page → {DEWA_PODCAST_URL}")
    try:
        page.goto(DEWA_PODCAST_URL, timeout=SCRAPER_TIMEOUT_MS, wait_until="networkidle")
        page.wait_for_timeout(SCRAPER_WAIT_AFTER_LOAD_MS)
    except PWTimeout:
        print("[scraper] Podcast page timed out — skipping")
        return []

    if debug:
        page.screenshot(path="debug_podcasts.png", full_page=True)
        Path("debug_podcasts.html").write_text(page.content(), encoding="utf-8")
        print("[debug] Saved debug_podcasts.png + debug_podcasts.html")

    # Auto-scroll
    for _ in range(5):
        page.evaluate("window.scrollBy(0, window.innerHeight)")
        page.wait_for_timeout(800)

    # Try "Load more"
    for _ in range(20):
        try:
            btn = page.query_selector(
                "button:has-text('Load more'), button:has-text('Show more'), "
                "a:has-text('Load more'), a:has-text('Next'), .pagination .next"
            )
            if btn and btn.is_visible():
                btn.click()
                page.wait_for_timeout(2000)
            else:
                break
        except Exception:
            break

    items = []
    for sel in PODCAST_ITEM_SELECTORS:
        found = page.query_selector_all(sel)
        if len(found) >= 2:
            items = found
            print(f"[scraper] Podcasts using selector '{sel}' → {len(found)} items")
            break

    results = []
    for item in items:
        try:
            title_el = first_match(item, TITLE_SELECTORS)
            if not title_el:
                continue
            title = title_el.inner_text().strip()
            href  = title_el.get_attribute("href") or ""
            if not href:
                link_el = item.query_selector("a")
                href = link_el.get_attribute("href") if link_el else ""
            url = normalise_url(href)

            if not title or not url:
                continue

            date_el = first_match(item, DATE_SELECTORS)
            raw_date = ""
            if date_el:
                raw_date = date_el.get_attribute("datetime") or date_el.inner_text()
            date = parse_date(raw_date)

            audio_el = item.query_selector("audio source, [src$='.mp3'], [href$='.mp3'], audio")
            audio_url = None
            if audio_el:
                audio_url = audio_el.get_attribute("src") or audio_el.get_attribute("href")
                audio_url = normalise_url(audio_url) if audio_url else None

            dur_el = item.query_selector(".duration, [class*='duration'], time")
            duration = dur_el.inner_text().strip() if dur_el else None

            results.append({
                "title": title,
                "url": url,
                "audio_url": audio_url,
                "date": date,
                "duration": duration,
            })
        except Exception as e:
            print(f"[scraper] Skipping podcast item: {e}")
            continue

    print(f"[scraper] Extracted {len(results)} podcasts")
    return results

def match_podcasts_to_prs(podcasts: list, pr_id_map: dict) -> list:
    """
    Try to link each podcast to a press release by URL fragment or title similarity.
    pr_id_map: { url: id }
    """
    matched = []
    for pod in podcasts:
        pr_id = None
        # Check if podcast URL matches any PR URL closely
        for pr_url, pid in pr_id_map.items():
            slug = pr_url.rstrip("/").split("/")[-1]
            if slug and slug in pod["url"]:
                pr_id = pid
                break
        matched.append({**pod, "pr_id": pr_id})
    return matched

# ── Main entry point ──────────────────────────────────────────────────────────

def run_scrape(debug: bool = False, headless: bool = None):
    if headless is None:
        headless = SCRAPER_HEADLESS
    if debug:
        headless = False  # always show browser in debug mode

    db.init_db()

    prs_saved = 0
    pods_saved = 0
    status = "success"
    notes = ""

    try:
        with sync_playwright() as pw:
            browser = pw.chromium.launch(
                headless=headless,
                args=["--disable-blink-features=AutomationControlled"],
            )
            context = browser.new_context(
                user_agent=(
                    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                    "AppleWebKit/537.36 (KHTML, like Gecko) "
                    "Chrome/122.0.0.0 Safari/537.36"
                ),
                viewport={"width": 1440, "height": 900},
            )
            page = context.new_page()

            # ── Press Releases ────────────────────────────────────────────────
            pr_data = scrape_press_releases(page, debug=debug)
            pr_id_map = {}
            for pr in pr_data:
                pid = db.upsert_press_release(
                    title=pr["title"],
                    url=pr["url"],
                    date=pr["date"],
                    category=pr["category"],
                    description=pr["description"],
                )
                pr_id_map[pr["url"]] = pid
                prs_saved += 1

            # ── Podcasts ──────────────────────────────────────────────────────
            pod_data = scrape_podcasts(page, debug=debug)
            pod_data = match_podcasts_to_prs(pod_data, pr_id_map)
            for pod in pod_data:
                db.upsert_podcast(
                    title=pod["title"],
                    url=pod["url"],
                    pr_id=pod.get("pr_id"),
                    audio_url=pod.get("audio_url"),
                    date=pod.get("date"),
                    duration=pod.get("duration"),
                )
                pods_saved += 1

            browser.close()

    except Exception as e:
        status = "error"
        notes = str(e)
        print(f"[scraper] ERROR: {e}")

    db.log_scrape(prs_saved, pods_saved, status, notes)
    print(f"\n[scraper] Done — {prs_saved} PRs, {pods_saved} podcasts saved. Status: {status}")
    if notes:
        print(f"[scraper] Notes: {notes}")

    return prs_saved, pods_saved

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="DEWA PR Scraper")
    parser.add_argument("--debug",   action="store_true", help="Save HTML/screenshots, open visible browser")
    parser.add_argument("--visible", action="store_true", help="Open visible browser without debug output")
    args = parser.parse_args()

    run_scrape(debug=args.debug, headless=not args.visible and SCRAPER_HEADLESS)


File created successfully at: /Users/khalidbinkalban/Projects/dewa-pr-tracker/scraper.py

"""
DEWA PR Tracker — Playwright-based scraper
Run directly:  python scraper.py
Debug mode:    python scraper.py --debug   (saves HTML + screenshot, opens browser)
"""
import argparse
import re
import sys
from datetime import datetime
from pathlib import Path

from playwright.sync_api import sync_playwright, TimeoutError as PWTimeout

import database as db
from config import (
    DEWA_NEWS_URL, DEWA_PODCAST_URL, DEWA_BASE_URL,
    SCRAPER_HEADLESS, SCRAPER_TIMEOUT_MS, SCRAPER_WAIT_AFTER_LOAD_MS,
)

# ── Selector candidates (tried in order until one matches) ───────────────────
PR_ITEM_SELECTORS = [
    "article",
    ".news-item",
    ".press-release-item",
    ".listing-item",
    ".media-item",
    "[class*='news']",
    "[class*='article']",
    "[class*='press']",
    ".container li",
    ".results-list li",
]

TITLE_SELECTORS  = ["h1", "h2", "h3", "h4", ".title", "[class*='title']", "a"]
DATE_SELECTORS   = ["time", ".date", "[class*='date']", "[datetime]", "span.label"]

PODCAST_ITEM_SELECTORS = [
    ".podcast-item",
    "article",
    ".media-item",
    "[class*='podcast']",
    ".listing-item",
]

# ── Utilities ─────────────────────────────────────────────────────────────────

def normalise_url(href: str) -> str:
    if not href:
        return ""
    href = href.strip()
    if href.startswith("http"):
        return href
    if href.startswith("/"):
        return DEWA_BASE_URL + href
    return href

def parse_date(raw: str) -> str:
    """Try to parse a messy date string into YYYY-MM-DD."""
    if not raw:
        return None
    raw = raw.strip()
    formats = [
        "%d %B %Y", "%B %d, %Y", "%d/%m/%Y",
        "%Y-%m-%d", "%d-%m-%Y", "%d %b %Y",
        "%B %Y",
    ]
    for fmt in formats:
        try:
            return datetime.strptime(raw, fmt).strftime("%Y-%m-%d")
        except ValueError:
            continue
    # Fallback: grab 4-digit year
    m = re.search(r"(\d{4})", raw)
    return m.group(1) + "-01-01" if m else None

def first_match(element, selectors):
    for sel in selectors:
        try:
            el = element.query_selector(sel)
            if el:
                return el
        except Exception:
            continue
    return None

# ── Core scraping logic ───────────────────────────────────────────────────────

def scrape_press_releases(page, debug: bool = False):
    print(f"[scraper] Loading news page → {DEWA_NEWS_URL}")
    page.goto(DEWA_NEWS_URL, timeout=SCRAPER_TIMEOUT_MS, wait_until="networkidle")
    page.wait_for_timeout(SCRAPER_WAIT_AFTER_LOAD_MS)

    if debug:
        page.screenshot(path="debug_news.png", full_page=True)
        Path("debug_news.html").write_text(page.content(), encoding="utf-8")
        print("[debug] Saved debug_news.png + debug_news.html")

    # Auto-scroll to trigger lazy-loading
    for _ in range(5):
        page.evaluate("window.scrollBy(0, window.innerHeight)")
        page.wait_for_timeout(800)

    # Try "Load more" / pagination buttons
    for _ in range(20):
        try:
            btn = page.query_selector(
                "button:has-text('Load more'), button:has-text('Show more'), "
                "a:has-text('Load more'), a:has-text('Next'), .pagination .next"
            )
            if btn and btn.is_visible():
                btn.click()
                page.wait_for_timeout(2000)
            else:
                break
        except Exception:
            break

    # Find the container with news items
    items = []
    for sel in PR_ITEM_SELECTORS:
        found = page.query_selector_all(sel)
        if len(found) >= 3:
            items = found
            print(f"[scraper] Using selector '{sel}' → {len(found)} items found")
            break

    if not items:
        print("[scraper] WARNING: No items found with known selectors. Run with --debug to inspect HTML.")
        return []

    results = []
    for item in items:
        try:
            # Title + URL
            title_el = first_match(item, TITLE_SELECTORS)
            if not title_el:
                continue
            title = title_el.inner_text().strip()
            href  = title_el.get_attribute("href") or ""
            if not href:
                link_el = item.query_selector("a")
                href = link_el.get_attribute("href") if link_el else ""
            url = normalise_url(href)

            if not title or not url:
                continue

            # Date
            date_el = first_match(item, DATE_SELECTORS)
            raw_date = ""
            if date_el:
                raw_date = date_el.get_attribute("datetime") or date_el.inner_text()
            date = parse_date(raw_date)

            # Category / description
            desc_el = item.query_selector("p, .description, .excerpt, .summary")
            description = desc_el.inner_text().strip() if desc_el else None

            cat_el = item.query_selector(".category, .tag, .label, [class*='categ']")
            category = cat_el.inner_text().strip() if cat_el else None

            results.append({
                "title": title,
                "url": url,
                "date": date,
                "category": category,
                "description": description,
            })
        except Exception as e:
            print(f"[scraper] Skipping item due to error: {e}")
            continue

    print(f"[scraper] Extracted {len(results)} press releases")
    return results

def scrape_podcasts(page, debug: bool = False):
    print(f"[scraper] Loading podcasts page → {DEWA_PODCAST_URL}")
    try:
        page.goto(DEWA_PODCAST_URL, timeout=SCRAPER_TIMEOUT_MS, wait_until="networkidle")
        page.wait_for_timeout(SCRAPER_WAIT_AFTER_LOAD_MS)
    except PWTimeout:
        print("[scraper] Podcast page timed out — skipping")
        return []

    if debug:
        page.screenshot(path="debug_podcasts.png", full_page=True)
        Path("debug_podcasts.html").write_text(page.content(), encoding="utf-8")
        print("[debug] Saved debug_podcasts.png + debug_podcasts.html")

    # Auto-scroll
    for _ in range(5):
        page.evaluate("window.scrollBy(0, window.innerHeight)")
        page.wait_for_timeout(800)

    # Try "Load more"
    for _ in range(20):
        try:
            btn = page.query_selector(
                "button:has-text('Load more'), button:has-text('Show more'), "
                "a:has-text('Load more'), a:has-text('Next'), .pagination .next"
            )
            if btn and btn.is_visible():
                btn.click()
                page.wait_for_timeout(2000)
            else:
                break
        except Exception:
            break

    items = []
    for sel in PODCAST_ITEM_SELECTORS:
        found = page.query_selector_all(sel)
        if len(found) >= 2:
            items = found
            print(f"[scraper] Podcasts using selector '{sel}' → {len(found)} items")
            break

    results = []
    for item in items:
        try:
            title_el = first_match(item, TITLE_SELECTORS)
            if not title_el:
                continue
            title = title_el.inner_text().strip()
            href  = title_el.get_attribute("href") or ""
            if not href:
                link_el = item.query_selector("a")
                href = link_el.get_attribute("href") if link_el else ""
            url = normalise_url(href)

            if not title or not url:
                continue

            date_el = first_match(item, DATE_SELECTORS)
            raw_date = ""
            if date_el:
                raw_date = date_el.get_attribute("datetime") or date_el.inner_text()
            date = parse_date(raw_date)

            audio_el = item.query_selector("audio source, [src$='.mp3'], [href$='.mp3'], audio")
            audio_url = None
            if audio_el:
                audio_url = audio_el.get_attribute("src") or audio_el.get_attribute("href")
                audio_url = normalise_url(audio_url) if audio_url else None

            dur_el = item.query_selector(".duration, [class*='duration'], time")
            duration = dur_el.inner_text().strip() if dur_el else None

            results.append({
                "title": title,
                "url": url,
                "audio_url": audio_url,
                "date": date,
                "duration": duration,
            })
        except Exception as e:
            print(f"[scraper] Skipping podcast item: {e}")
            continue

    print(f"[scraper] Extracted {len(results)} podcasts")
    return results

def match_podcasts_to_prs(podcasts: list, pr_id_map: dict) -> list:
    """
    Try to link each podcast to a press release by URL fragment or title similarity.
    pr_id_map: { url: id }
    """
    matched = []
    for pod in podcasts:
        pr_id = None
        # Check if podcast URL matches any PR URL closely
        for pr_url, pid in pr_id_map.items():
            slug = pr_url.rstrip("/").split("/")[-1]
            if slug and slug in pod["url"]:
                pr_id = pid
                break
        matched.append({**pod, "pr_id": pr_id})
    return matched

# ── Main entry point ──────────────────────────────────────────────────────────

def run_scrape(debug: bool = False, headless: bool = None):
    if headless is None:
        headless = SCRAPER_HEADLESS
    if debug:
        headless = False  # always show browser in debug mode

    db.init_db()

    prs_saved = 0
    pods_saved = 0
    status = "success"
    notes = ""

    try:
        with sync_playwright() as pw:
            browser = pw.chromium.launch(
                headless=headless,
                args=["--disable-blink-features=AutomationControlled"],
            )
            context = browser.new_context(
                user_agent=(
                    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                    "AppleWebKit/537.36 (KHTML, like Gecko) "
                    "Chrome/122.0.0.0 Safari/537.36"
                ),
                viewport={"width": 1440, "height": 900},
            )
            page = context.new_page()

            # ── Press Releases ────────────────────────────────────────────────
            pr_data = scrape_press_releases(page, debug=debug)
            pr_id_map = {}
            for pr in pr_data:
                pid = db.upsert_press_release(
                    title=pr["title"],
                    url=pr["url"],
                    date=pr["date"],
                    category=pr["category"],
                    description=pr["description"],
                )
                pr_id_map[pr["url"]] = pid
                prs_saved += 1

            # ── Podcasts ──────────────────────────────────────────────────────
            pod_data = scrape_podcasts(page, debug=debug)
            pod_data = match_podcasts_to_prs(pod_data, pr_id_map)
            for pod in pod_data:
                db.upsert_podcast(
                    title=pod["title"],
                    url=pod["url"],
                    pr_id=pod.get("pr_id"),
                    audio_url=pod.get("audio_url"),
                    date=pod.get("date"),
                    duration=pod.get("duration"),
                )
                pods_saved += 1

            browser.close()

    except Exception as e:
        status = "error"
        notes = str(e)
        print(f"[scraper] ERROR: {e}")

    db.log_scrape(prs_saved, pods_saved, status, notes)
    print(f"\n[scraper] Done — {prs_saved} PRs, {pods_saved} podcasts saved. Status: {status}")
    if notes:
        print(f"[scraper] Notes: {notes}")

    return prs_saved, pods_saved

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="DEWA PR Scraper")
    parser.add_argument("--debug",   action="store_true", help="Save HTML/screenshots, open visible browser")
    parser.add_argument("--visible", action="store_true", help="Open visible browser without debug output")
    args = parser.parse_args()

    run_scrape(debug=args.debug, headless=not args.visible and SCRAPER_HEADLESS)


"""
DEWA PR Tracker — Streamlit Dashboard
Run:  streamlit run dashboard.py
"""
import math
from datetime import datetime, timedelta

import pandas as pd
import plotly.express as px
import plotly.graph_objects as go
import streamlit as st

import database as db
from config import (
    COST_PER_PODCAST_OLD_AED,
    NOTEBOOKLM_MONTHLY_USD,
    USD_TO_AED,
    DASHBOARD_TITLE,
    PAGE_SIZE,
)

# ── Page setup ────────────────────────────────────────────────────────────────
st.set_page_config(
    page_title=DASHBOARD_TITLE,
    page_icon="📰",
    layout="wide",
    initial_sidebar_state="expanded",
)

# ── Custom CSS ────────────────────────────────────────────────────────────────
st.markdown("""

""", unsafe_allow_html=True)

# ── Data loading (cached 5 min) ───────────────────────────────────────────────
@st.cache_data(ttl=300)
def load_data():
    db.init_db()
    prs   = [dict(r) for r in db.get_all_press_releases()]
    pods  = [dict(r) for r in db.get_all_podcasts()]
    stats = db.get_stats()
    log   = [dict(r) for r in db.get_scrape_log()]
    return prs, pods, stats, log

prs, pods, stats, scrape_log = load_data()
df_pr  = pd.DataFrame(prs)
df_pod = pd.DataFrame(pods)

if not df_pr.empty:
    df_pr["date"] = pd.to_datetime(df_pr["date"], errors="coerce")
    df_pr["week"]  = df_pr["date"].dt.to_period("W").astype(str)
    df_pr["month"] = df_pr["date"].dt.to_period("M").astype(str)
    df_pr["year"]  = df_pr["date"].dt.year

if not df_pod.empty:
    df_pod["date"] = pd.to_datetime(df_pod["date"], errors="coerce")
    df_pod["month"] = df_pod["date"].dt.to_period("M").astype(str)

# ── Sidebar ───────────────────────────────────────────────────────────────────
with st.sidebar:
    st.image("https://www.dewa.gov.ae/~/media/Images/Header/DEWA-Logo.ashx", width=160)
    st.title("Filters")

    # Date range
    if not df_pr.empty and df_pr["date"].notna().any():
        min_date = df_pr["date"].min().date()
        max_date = df_pr["date"].max().date()
    else:
        min_date = datetime(2020, 1, 1).date()
        max_date = datetime.today().date()

    date_from, date_to = st.date_input(
        "Date range",
        value=(min_date, max_date),
        min_value=min_date,
        max_value=max_date,
    )

    # Category filter
    categories = ["All"]
    if not df_pr.empty and "category" in df_pr.columns:
        cats = df_pr["category"].dropna().unique().tolist()
        categories += sorted(cats)
    selected_cat = st.selectbox("Category", categories)

    # Podcast filter
    has_podcast_filter = st.checkbox("Show only PRs with podcast", value=False)

    st.divider()
    if st.button("🔄 Refresh data", use_container_width=True):
        st.cache_data.clear()
        st.rerun()

    if st.button("▶ Run scraper now", use_container_width=True):
        with st.spinner("Scraping DEWA website…"):
            try:
                from scraper import run_scrape
                prs_n, pods_n = run_scrape()
                st.success(f"Done! {prs_n} PRs, {pods_n} podcasts saved.")
                st.cache_data.clear()
                st.rerun()
            except Exception as e:
                st.error(f"Scraper error: {e}")

    st.divider()
    st.caption(f"Last scraped: {stats.get('last_scraped', 'Never') or 'Never'}")

# ── Apply filters ─────────────────────────────────────────────────────────────
filtered = df_pr.copy()
if not filtered.empty:
    if filtered["date"].notna().any():
        filtered = filtered[
            (filtered["date"].dt.date >= date_from) &
            (filtered["date"].dt.date <= date_to)
        ]
    if selected_cat != "All":
        filtered = filtered[filtered["category"] == selected_cat]
    if has_podcast_filter:
        filtered = filtered[filtered["podcast_count"] > 0]

# ═══════════════════════════════════════════════════════════════════════════════
# HEADER
# ═══════════════════════════════════════════════════════════════════════════════
st.title(f"📰 {DASHBOARD_TITLE}")
st.markdown(f"*Tracking DEWA press releases and podcast ROI — {datetime.today().strftime('%d %B %Y')}*")
st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 1 — Key Metrics
# ═══════════════════════════════════════════════════════════════════════════════
col1, col2, col3, col4, col5 = st.columns(5)

total_prs  = len(filtered)
total_pods = stats.get("total_podcasts", 0)

today = datetime.today().date()
week_start  = today - timedelta(days=today.weekday())
month_start = today.replace(day=1)

if not filtered.empty and filtered["date"].notna().any():
    prs_today   = len(filtered[filtered["date"].dt.date == today])
    prs_week    = len(filtered[filtered["date"].dt.date >= week_start])
    prs_month   = len(filtered[filtered["date"].dt.date >= month_start])
else:
    prs_today = prs_week = prs_month = 0

with col1:
    st.metric("Total Press Releases", f"{total_prs:,}")
with col2:
    st.metric("Total Podcasts", f"{total_pods:,}")
with col3:
    st.metric("Published Today", prs_today)
with col4:
    st.metric("This Week", prs_week)
with col5:
    st.metric("This Month", prs_month)

st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 2 — ROI Calculator
# ═══════════════════════════════════════════════════════════════════════════════
st.subheader("💰 Podcast ROI Calculator")

notebooklm_aed = NOTEBOOKLM_MONTHLY_USD * USD_TO_AED

# Calculate months since NotebookLM adoption (user can adjust in the UI)
c_left, c_right = st.columns([2, 1])
with c_right:
    months_using = st.number_input(
        "Months using NotebookLM", min_value=1, max_value=60, value=6
    )
    podcasts_in_period = st.number_input(
        "Podcasts produced in that period", min_value=0, max_value=500,
        value=total_pods
    )

old_cost  = podcasts_in_period * COST_PER_PODCAST_OLD_AED
new_cost  = months_using * notebooklm_aed
savings   = old_cost - new_cost
roi_pct   = (savings / old_cost * 100) if old_cost > 0 else 0

with c_left:
    r1, r2, r3 = st.columns(3)
    with r1:
        st.markdown("**Before — Traditional Production**")
        st.markdown(f"AED {old_cost:,.0f}", unsafe_allow_html=True)
        st.caption(f"{podcasts_in_period} podcast(s) × AED {COST_PER_PODCAST_OLD_AED:,} each")

    with r2:
        st.markdown("**After — NotebookLM**")
        st.markdown(f"AED {new_cost:,.0f}", unsafe_allow_html=True)
        st.caption(f"{months_using} month(s) × AED {notebooklm_aed:.0f}/mo (${NOTEBOOKLM_MONTHLY_USD}/mo)")

    with r3:
        st.markdown("**Net Savings**")
        st.markdown(f"AED {savings:,.0f}", unsafe_allow_html=True)
        st.caption(f"ROI: {roi_pct:.1f}%")

    # ROI gauge
    fig_gauge = go.Figure(go.Indicator(
        mode="gauge+number+delta",
        value=roi_pct,
        title={"text": "ROI %"},
        delta={"reference": 0},
        gauge={
            "axis": {"range": [0, 100]},
            "bar": {"color": "#00c853"},
            "steps": [
                {"range": [0, 50], "color": "#1e2a3a"},
                {"range": [50, 80], "color": "#0d3b2e"},
                {"range": [80, 100], "color": "#0a4a35"},
            ],
            "threshold": {"line": {"color": "gold", "width": 4}, "value": 90},
        },
        number={"suffix": "%"},
    ))
    fig_gauge.update_layout(height=220, margin=dict(t=30, b=0, l=20, r=20),
                            paper_bgcolor="rgba(0,0,0,0)", font_color="white")
    st.plotly_chart(fig_gauge, use_container_width=True)

st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 3 — Publication Charts
# ═══════════════════════════════════════════════════════════════════════════════
st.subheader("📊 Publication Trends")
tab_daily, tab_weekly, tab_monthly = st.tabs(["Daily", "Weekly", "Monthly"])

def make_bar(df_grouped, x, y, title, color="#00b4d8"):
    fig = px.bar(df_grouped, x=x, y=y, title=title,
                 color_discrete_sequence=[color])
    fig.update_layout(
        paper_bgcolor="rgba(0,0,0,0)",
        plot_bgcolor="rgba(0,0,0,0)",
        font_color="white",
        xaxis=dict(showgrid=False),
        yaxis=dict(gridcolor="#2a3a4a"),
        margin=dict(t=40, b=10),
    )
    return fig

with tab_daily:
    if not filtered.empty and filtered["date"].notna().any():
        daily = filtered.groupby(filtered["date"].dt.date).size().reset_index(name="count")
        daily.columns = ["date", "count"]
        st.plotly_chart(make_bar(daily, "date", "count", "Press Releases per Day"), use_container_width=True)
    else:
        st.info("No date data available yet. Run the scraper first.")

with tab_weekly:
    if not filtered.empty and "week" in filtered.columns and filtered["date"].notna().any():
        weekly = filtered.groupby("week").size().reset_index(name="count")
        st.plotly_chart(make_bar(weekly, "week", "count", "Press Releases per Week", "#4cc9f0"), use_container_width=True)
    else:
        st.info("No data yet.")

with tab_monthly:
    if not filtered.empty and "month" in filtered.columns and filtered["date"].notna().any():
        monthly = filtered.groupby("month").size().reset_index(name="count")
        st.plotly_chart(make_bar(monthly, "month", "count", "Press Releases per Month", "#7209b7"), use_container_width=True)
    else:
        st.info("No data yet.")

st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 4 — Press Releases Table
# ═══════════════════════════════════════════════════════════════════════════════
st.subheader("📋 Press Releases")

search = st.text_input("Search press releases", placeholder="Type to filter by title…")
if search:
    display = filtered[filtered["title"].str.contains(search, case=False, na=False)]
else:
    display = filtered.copy()

total_pages = max(1, math.ceil(len(display) / PAGE_SIZE))
page_num = st.number_input("Page", min_value=1, max_value=total_pages, value=1, step=1)
start = (page_num - 1) * PAGE_SIZE
end   = start + PAGE_SIZE
page_data = display.iloc[start:end]

st.caption(f"Showing {start+1}–{min(end, len(display))} of {len(display)} press releases (page {page_num}/{total_pages})")

for _, row in page_data.iterrows():
    with st.container():
        tc, dc, pc = st.columns([5, 2, 1])
        with tc:
            url   = row.get("url", "")
            title = row.get("title", "—")
            st.markdown(f"**[{title}]({url})**" if url else f"**{title}**")
            if row.get("description"):
                st.caption(row["description"][:160] + "…" if len(str(row["description"])) > 160 else row["description"])
        with dc:
            d = row.get("date")
            st.markdown(f"`{d.strftime('%d %b %Y') if pd.notna(d) else '—'}`")
            if row.get("category"):
                st.caption(row["category"])
        with pc:
            pc_count = row.get("podcast_count", 0)
            if pc_count:
                st.markdown(f"🎙 {pc_count}")
            else:
                st.markdown("—")
    st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 5 — Podcasts Table
# ═══════════════════════════════════════════════════════════════════════════════
if not df_pod.empty:
    st.subheader("🎙 Podcasts")
    for _, pod in df_pod.iterrows():
        with st.expander(f"{pod.get('title', '—')}  |  {pod.get('date', '')}"):
            col_a, col_b = st.columns([3, 1])
            with col_a:
                if pod.get("pr_title"):
                    st.markdown(f"**Linked PR:** {pod['pr_title']}")
                if pod.get("url"):
                    st.markdown(f"[Open podcast page]({pod['url']})")
                if pod.get("audio_url"):
                    st.audio(pod["audio_url"])
            with col_b:
                if pod.get("duration"):
                    st.metric("Duration", pod["duration"])
                cost_saved = COST_PER_PODCAST_OLD_AED
                st.metric("Saved vs old cost", f"AED {cost_saved:,}")

st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 6 — Scrape Log
# ═══════════════════════════════════════════════════════════════════════════════
with st.expander("🔍 Scrape history"):
    if scrape_log:
        df_log = pd.DataFrame(scrape_log)
        st.dataframe(df_log[["ran_at", "prs_found", "pods_found", "status", "notes"]],
                     use_container_width=True)
    else:
        st.info("No scrape runs recorded yet.")

st.caption("DEWA PR Tracker · Built with Playwright + SQLite + Streamlit")


File created successfully at: /Users/khalidbinkalban/Projects/dewa-pr-tracker/dashboard.py

"""
DEWA PR Tracker — Streamlit Dashboard
Run:  streamlit run dashboard.py
"""
import math
from datetime import datetime, timedelta

import pandas as pd
import plotly.express as px
import plotly.graph_objects as go
import streamlit as st

import database as db
from config import (
    COST_PER_PODCAST_OLD_AED,
    NOTEBOOKLM_MONTHLY_USD,
    USD_TO_AED,
    DASHBOARD_TITLE,
    PAGE_SIZE,
)

# ── Page setup ────────────────────────────────────────────────────────────────
st.set_page_config(
    page_title=DASHBOARD_TITLE,
    page_icon="📰",
    layout="wide",
    initial_sidebar_state="expanded",
)

# ── Custom CSS ────────────────────────────────────────────────────────────────
st.markdown("""

""", unsafe_allow_html=True)

# ── Data loading (cached 5 min) ───────────────────────────────────────────────
@st.cache_data(ttl=300)
def load_data():
    db.init_db()
    prs   = [dict(r) for r in db.get_all_press_releases()]
    pods  = [dict(r) for r in db.get_all_podcasts()]
    stats = db.get_stats()
    log   = [dict(r) for r in db.get_scrape_log()]
    return prs, pods, stats, log

prs, pods, stats, scrape_log = load_data()
df_pr  = pd.DataFrame(prs)
df_pod = pd.DataFrame(pods)

if not df_pr.empty:
    df_pr["date"] = pd.to_datetime(df_pr["date"], errors="coerce")
    df_pr["week"]  = df_pr["date"].dt.to_period("W").astype(str)
    df_pr["month"] = df_pr["date"].dt.to_period("M").astype(str)
    df_pr["year"]  = df_pr["date"].dt.year

if not df_pod.empty:
    df_pod["date"] = pd.to_datetime(df_pod["date"], errors="coerce")
    df_pod["month"] = df_pod["date"].dt.to_period("M").astype(str)

# ── Sidebar ───────────────────────────────────────────────────────────────────
with st.sidebar:
    st.image("https://www.dewa.gov.ae/~/media/Images/Header/DEWA-Logo.ashx", width=160)
    st.title("Filters")

    # Date range
    if not df_pr.empty and df_pr["date"].notna().any():
        min_date = df_pr["date"].min().date()
        max_date = df_pr["date"].max().date()
    else:
        min_date = datetime(2020, 1, 1).date()
        max_date = datetime.today().date()

    date_from, date_to = st.date_input(
        "Date range",
        value=(min_date, max_date),
        min_value=min_date,
        max_value=max_date,
    )

    # Category filter
    categories = ["All"]
    if not df_pr.empty and "category" in df_pr.columns:
        cats = df_pr["category"].dropna().unique().tolist()
        categories += sorted(cats)
    selected_cat = st.selectbox("Category", categories)

    # Podcast filter
    has_podcast_filter = st.checkbox("Show only PRs with podcast", value=False)

    st.divider()
    if st.button("🔄 Refresh data", use_container_width=True):
        st.cache_data.clear()
        st.rerun()

    if st.button("▶ Run scraper now", use_container_width=True):
        with st.spinner("Scraping DEWA website…"):
            try:
                from scraper import run_scrape
                prs_n, pods_n = run_scrape()
                st.success(f"Done! {prs_n} PRs, {pods_n} podcasts saved.")
                st.cache_data.clear()
                st.rerun()
            except Exception as e:
                st.error(f"Scraper error: {e}")

    st.divider()
    st.caption(f"Last scraped: {stats.get('last_scraped', 'Never') or 'Never'}")

# ── Apply filters ─────────────────────────────────────────────────────────────
filtered = df_pr.copy()
if not filtered.empty:
    if filtered["date"].notna().any():
        filtered = filtered[
            (filtered["date"].dt.date >= date_from) &
            (filtered["date"].dt.date <= date_to)
        ]
    if selected_cat != "All":
        filtered = filtered[filtered["category"] == selected_cat]
    if has_podcast_filter:
        filtered = filtered[filtered["podcast_count"] > 0]

# ═══════════════════════════════════════════════════════════════════════════════
# HEADER
# ═══════════════════════════════════════════════════════════════════════════════
st.title(f"📰 {DASHBOARD_TITLE}")
st.markdown(f"*Tracking DEWA press releases and podcast ROI — {datetime.today().strftime('%d %B %Y')}*")
st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 1 — Key Metrics
# ═══════════════════════════════════════════════════════════════════════════════
col1, col2, col3, col4, col5 = st.columns(5)

total_prs  = len(filtered)
total_pods = stats.get("total_podcasts", 0)

today = datetime.today().date()
week_start  = today - timedelta(days=today.weekday())
month_start = today.replace(day=1)

if not filtered.empty and filtered["date"].notna().any():
    prs_today   = len(filtered[filtered["date"].dt.date == today])
    prs_week    = len(filtered[filtered["date"].dt.date >= week_start])
    prs_month   = len(filtered[filtered["date"].dt.date >= month_start])
else:
    prs_today = prs_week = prs_month = 0

with col1:
    st.metric("Total Press Releases", f"{total_prs:,}")
with col2:
    st.metric("Total Podcasts", f"{total_pods:,}")
with col3:
    st.metric("Published Today", prs_today)
with col4:
    st.metric("This Week", prs_week)
with col5:
    st.metric("This Month", prs_month)

st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 2 — ROI Calculator
# ═══════════════════════════════════════════════════════════════════════════════
st.subheader("💰 Podcast ROI Calculator")

notebooklm_aed = NOTEBOOKLM_MONTHLY_USD * USD_TO_AED

# Calculate months since NotebookLM adoption (user can adjust in the UI)
c_left, c_right = st.columns([2, 1])
with c_right:
    months_using = st.number_input(
        "Months using NotebookLM", min_value=1, max_value=60, value=6
    )
    podcasts_in_period = st.number_input(
        "Podcasts produced in that period", min_value=0, max_value=500,
        value=total_pods
    )

old_cost  = podcasts_in_period * COST_PER_PODCAST_OLD_AED
new_cost  = months_using * notebooklm_aed
savings   = old_cost - new_cost
roi_pct   = (savings / old_cost * 100) if old_cost > 0 else 0

with c_left:
    r1, r2, r3 = st.columns(3)
    with r1:
        st.markdown("**Before — Traditional Production**")
        st.markdown(f"AED {old_cost:,.0f}", unsafe_allow_html=True)
        st.caption(f"{podcasts_in_period} podcast(s) × AED {COST_PER_PODCAST_OLD_AED:,} each")

    with r2:
        st.markdown("**After — NotebookLM**")
        st.markdown(f"AED {new_cost:,.0f}", unsafe_allow_html=True)
        st.caption(f"{months_using} month(s) × AED {notebooklm_aed:.0f}/mo (${NOTEBOOKLM_MONTHLY_USD}/mo)")

    with r3:
        st.markdown("**Net Savings**")
        st.markdown(f"AED {savings:,.0f}", unsafe_allow_html=True)
        st.caption(f"ROI: {roi_pct:.1f}%")

    # ROI gauge
    fig_gauge = go.Figure(go.Indicator(
        mode="gauge+number+delta",
        value=roi_pct,
        title={"text": "ROI %"},
        delta={"reference": 0},
        gauge={
            "axis": {"range": [0, 100]},
            "bar": {"color": "#00c853"},
            "steps": [
                {"range": [0, 50], "color": "#1e2a3a"},
                {"range": [50, 80], "color": "#0d3b2e"},
                {"range": [80, 100], "color": "#0a4a35"},
            ],
            "threshold": {"line": {"color": "gold", "width": 4}, "value": 90},
        },
        number={"suffix": "%"},
    ))
    fig_gauge.update_layout(height=220, margin=dict(t=30, b=0, l=20, r=20),
                            paper_bgcolor="rgba(0,0,0,0)", font_color="white")
    st.plotly_chart(fig_gauge, use_container_width=True)

st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 3 — Publication Charts
# ═══════════════════════════════════════════════════════════════════════════════
st.subheader("📊 Publication Trends")
tab_daily, tab_weekly, tab_monthly = st.tabs(["Daily", "Weekly", "Monthly"])

def make_bar(df_grouped, x, y, title, color="#00b4d8"):
    fig = px.bar(df_grouped, x=x, y=y, title=title,
                 color_discrete_sequence=[color])
    fig.update_layout(
        paper_bgcolor="rgba(0,0,0,0)",
        plot_bgcolor="rgba(0,0,0,0)",
        font_color="white",
        xaxis=dict(showgrid=False),
        yaxis=dict(gridcolor="#2a3a4a"),
        margin=dict(t=40, b=10),
    )
    return fig

with tab_daily:
    if not filtered.empty and filtered["date"].notna().any():
        daily = filtered.groupby(filtered["date"].dt.date).size().reset_index(name="count")
        daily.columns = ["date", "count"]
        st.plotly_chart(make_bar(daily, "date", "count", "Press Releases per Day"), use_container_width=True)
    else:
        st.info("No date data available yet. Run the scraper first.")

with tab_weekly:
    if not filtered.empty and "week" in filtered.columns and filtered["date"].notna().any():
        weekly = filtered.groupby("week").size().reset_index(name="count")
        st.plotly_chart(make_bar(weekly, "week", "count", "Press Releases per Week", "#4cc9f0"), use_container_width=True)
    else:
        st.info("No data yet.")

with tab_monthly:
    if not filtered.empty and "month" in filtered.columns and filtered["date"].notna().any():
        monthly = filtered.groupby("month").size().reset_index(name="count")
        st.plotly_chart(make_bar(monthly, "month", "count", "Press Releases per Month", "#7209b7"), use_container_width=True)
    else:
        st.info("No data yet.")

st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 4 — Press Releases Table
# ═══════════════════════════════════════════════════════════════════════════════
st.subheader("📋 Press Releases")

search = st.text_input("Search press releases", placeholder="Type to filter by title…")
if search:
    display = filtered[filtered["title"].str.contains(search, case=False, na=False)]
else:
    display = filtered.copy()

total_pages = max(1, math.ceil(len(display) / PAGE_SIZE))
page_num = st.number_input("Page", min_value=1, max_value=total_pages, value=1, step=1)
start = (page_num - 1) * PAGE_SIZE
end   = start + PAGE_SIZE
page_data = display.iloc[start:end]

st.caption(f"Showing {start+1}–{min(end, len(display))} of {len(display)} press releases (page {page_num}/{total_pages})")

for _, row in page_data.iterrows():
    with st.container():
        tc, dc, pc = st.columns([5, 2, 1])
        with tc:
            url   = row.get("url", "")
            title = row.get("title", "—")
            st.markdown(f"**[{title}]({url})**" if url else f"**{title}**")
            if row.get("description"):
                st.caption(row["description"][:160] + "…" if len(str(row["description"])) > 160 else row["description"])
        with dc:
            d = row.get("date")
            st.markdown(f"`{d.strftime('%d %b %Y') if pd.notna(d) else '—'}`")
            if row.get("category"):
                st.caption(row["category"])
        with pc:
            pc_count = row.get("podcast_count", 0)
            if pc_count:
                st.markdown(f"🎙 {pc_count}")
            else:
                st.markdown("—")
    st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 5 — Podcasts Table
# ═══════════════════════════════════════════════════════════════════════════════
if not df_pod.empty:
    st.subheader("🎙 Podcasts")
    for _, pod in df_pod.iterrows():
        with st.expander(f"{pod.get('title', '—')}  |  {pod.get('date', '')}"):
            col_a, col_b = st.columns([3, 1])
            with col_a:
                if pod.get("pr_title"):
                    st.markdown(f"**Linked PR:** {pod['pr_title']}")
                if pod.get("url"):
                    st.markdown(f"[Open podcast page]({pod['url']})")
                if pod.get("audio_url"):
                    st.audio(pod["audio_url"])
            with col_b:
                if pod.get("duration"):
                    st.metric("Duration", pod["duration"])
                cost_saved = COST_PER_PODCAST_OLD_AED
                st.metric("Saved vs old cost", f"AED {cost_saved:,}")

st.divider()

# ═══════════════════════════════════════════════════════════════════════════════
# ROW 6 — Scrape Log
# ═══════════════════════════════════════════════════════════════════════════════
with st.expander("🔍 Scrape history"):
    if scrape_log:
        df_log = pd.DataFrame(scrape_log)
        st.dataframe(df_log[["ran_at", "prs_found", "pods_found", "status", "notes"]],
                     use_container_width=True)
    else:
        st.info("No scrape runs recorded yet.")

st.caption("DEWA PR Tracker · Built with Playwright + SQLite + Streamlit")


"""
DEWA PR Tracker — Scheduler
Runs the scraper automatically every day at 08:00 AM.

Run:  python scheduler.py
"""
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.triggers.cron import CronTrigger
from datetime import datetime

from scraper import run_scrape

def job():
    print(f"\n[scheduler] {datetime.now().isoformat()} — Starting scheduled scrape…")
    run_scrape()
    print(f"[scheduler] Scrape complete.")

if __name__ == "__main__":
    scheduler = BlockingScheduler(timezone="Asia/Dubai")
    scheduler.add_job(job, CronTrigger(hour=8, minute=0))
    print("[scheduler] Started — scraper will run daily at 08:00 Asia/Dubai time.")
    print("[scheduler] Press Ctrl+C to stop.")
    try:
        scheduler.start()
    except (KeyboardInterrupt, SystemExit):
        print("[scheduler] Stopped.")


File created successfully at: /Users/khalidbinkalban/Projects/dewa-pr-tracker/scheduler.py

"""
DEWA PR Tracker — Scheduler
Runs the scraper automatically every day at 08:00 AM.

Run:  python scheduler.py
"""
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.triggers.cron import CronTrigger
from datetime import datetime

from scraper import run_scrape

def job():
    print(f"\n[scheduler] {datetime.now().isoformat()} — Starting scheduled scrape…")
    run_scrape()
    print(f"[scheduler] Scrape complete.")

if __name__ == "__main__":
    scheduler = BlockingScheduler(timezone="Asia/Dubai")
    scheduler.add_job(job, CronTrigger(hour=8, minute=0))
    print("[scheduler] Started — scraper will run daily at 08:00 Asia/Dubai time.")
    print("[scheduler] Press Ctrl+C to stop.")
    try:
        scheduler.start()
    except (KeyboardInterrupt, SystemExit):
        print("[scheduler] Stopped.")


playwright>=1.43.0
streamlit>=1.35.0
plotly>=5.22.0
pandas>=2.2.0
apscheduler>=3.10.4


File created successfully at: /Users/khalidbinkalban/Projects/dewa-pr-tracker/requirements.txt

playwright>=1.43.0
streamlit>=1.35.0
plotly>=5.22.0
pandas>=2.2.0
apscheduler>=3.10.4


Now let's install everything and run the first scrape:


Bash

cd /Users/khalidbinkalban/Projects/dewa-pr-tracker && python3 -m venv venv && source venv/bin/activate && pip install -q -r requirements.txt 2>&1 | tail -5

Create venv and install dependencies


WARNING: You are using pip version 21.2.4; however, version 26.0.1 is available.
You should consider upgrading via the '/Users/khalidbinkalban/Projects/dewa-pr-tracker/venv/bin/python3 -m pip install --upgrade pip' command.


source venv/bin/activate && playwright install chromium 2>&1 | tail -5

Install Playwright Chromium browser


source venv/bin/activate && python scraper.py --debug 2>&1

Run scraper in debug mode to inspect DEWA page structure


[scraper] Loading news page → https://www.dewa.gov.ae/en/about-us/media-publications/latest-news
[debug] Saved debug_news.png + debug_news.html
[scraper] Using selector '[class*='news']' → 8 items found
[scraper] Extracted 8 press releases
[scraper] Loading podcasts page → https://www.dewa.gov.ae/en/about-us/media-publications/podcasts
[debug] Saved debug_podcasts.png + debug_podcasts.html
[scraper] Extracted 0 podcasts

[scraper] Done — 8 PRs, 0 podcasts saved. Status: success


Good start — it found 8 PRs but pagination isn't working and podcasts got 0. Let me inspect the saved HTML to fix the exact selectors.


python3 -c "
from pathlib import Path
html = Path('debug_news.html').read_text()
# Find news item patterns
import re
# Look for links in news sections
matches = re.findall(r'<a[^>]+href=\"([^\"]+latest-news[^\"]*|[^\"]+press[^\"]*|[^\"]+media[^\"]*)[^>]*>([^<]+)<', html)
for href, text in matches[:20]:
    print(repr(href[:80]), '|', repr(text[:60]))
"

Inspect news links in debug HTML


'/-/media/Files/Customer/Scrap-Sales-Registration-and-Update-Guide.ashx' | 'Scrap Sales – Registration & Update Profile Guide'
'/-/media/Files/Builder/Jointer-Certification--Testing-Services--Registration--Up' | 'Jointer Certification & Testing Services – Registration '
'/-/media/Files/Consultants-and-Contractors/Contractors-and-Consultants-Guide.ash' | 'Consultants & Contractor Law no. (6) Guide'
'/-/media/DEWA_Regulations_for_Electrical_Installations_2017_Edition.ashx' | 'Electricity Installations Regulations'
'/-/media/Files/Consultants-and-Contractors/SAFETY-AND-PROTECTION-GUIDELINES.ashx' | 'Safety and Protection Guidelines - Distribution MV Network '
'/-/media/Files/Customer/Scrap-Sales-Registration-and-Update-Guide.ashx' | 'Scrap Sales – Registration & Update Profile Guide'
'/-/media/Files/Builder/Jointer-Certification--Testing-Services--Registration--Up' | 'Jointer Certification & Testing Services – Registration '
'/-/media/Files/Supplier-Enrollment/Company_Profile_Update.ashx' | 'Company Profile Update Guide'
'/-/media/Files/Suppliers-and-Partners/DEWACodeofConductEthicsforSuppliersContrac' | 'Suppliers code of Conduct & Ethics'
'/-/media/Files/Suppliers-and-Partners/Security-E-Pass-Manual.ashx' | 'Security E-Pass Manual'
'/-/media/Files/HS-code-collection-Form.ashx' | 'HS code collection Form'
'/en/about-us/media-publications/latest-news' | 'Latest News'
'/en/about-us/media-publications/press-pack' | 'Media Centre'
'/en/about-us/media-publications/photo-gallery' | 'Photo Gallery'
'/en/about-us/media-publications/video-album-gallery' | 'Video Gallery'
'/ar-AE/about-us/media-publications/latest-news' | '\n            العربية\n        '
'/ar-AE/about-us/media-publications/latest-news' | '\n            العربية\n        '
'/en/about-us/media-publications/latest-news/2026/3/dewa-supports-dubai-autism' | '\n\n                '
'/en/about-us/media-publications/latest-news/2026/3/empower-annual-general-meetin' | '\n\n                '
'/en/about-us/media-publications/latest-news/2026/3/dewas-damj-initiative-support' | '\n\n                '


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Find the actual news article structure
# Look for the news list items with /latest-news/ in the href
pattern = r'href=\"(/en/about-us/media-publications/latest-news/\d{4}/\d+/[^\"]+)\"'
urls = re.findall(pattern, html)
print('News URLs found:', len(urls))
for u in urls[:10]:
    print(' ', u)

# Find what wraps these links
for url in urls[:3]:
    idx = html.find(url)
    snippet = html[max(0,idx-300):idx+300]
    print('\n--- snippet around URL ---')
    print(snippet[:600])
    print('---')
"

Extract news URL pattern and surrounding HTML structure


News URLs found: 8
  /en/about-us/media-publications/latest-news/2026/3/dewa-supports-dubai-autism
  /en/about-us/media-publications/latest-news/2026/3/empower-annual-general-meeting
  /en/about-us/media-publications/latest-news/2026/3/dewas-damj-initiative-supports-empowerment
  /en/about-us/media-publications/latest-news/2026/3/quote-by-he-saeed-mohammed
  /en/about-us/media-publications/latest-news/2026/3/dewas-ownek-platform-streamlines
  /en/about-us/media-publications/latest-news/2026/3/dewa-promotes-the-noble-values
  /en/about-us/media-publications/latest-news/2026/3/quote-by-he-saeed-mohammed-al-tayer
  /en/about-us/media-publications/latest-news/2026/3/dewa-contributes-aed20-million

--- snippet around URL ---

                <div class="m9-teaser-image tooltipstered" data-alt="DEWA supports Dubai Autism Center’s 20th Autism Awareness Campaign as Sustainab
---

--- snippet around URL ---

                <div class="m9-teaser-image tooltipstered" data-alt="Empower Annual General Meeting approves cash dividends of AED 437.5 million
---

--- snippet around URL ---

                <div class="m9-teaser-image tooltipstered" data-alt="DEWA's Damj initiative supports People of Determination" style=
---


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Look for date elements inside the teaser
# Find date patterns near the teaser items
idx = html.find('m9-teaser--news_all')
snippet = html[idx:idx+1500]
print(snippet)
"

Inspect full teaser item HTML for date and title elements


m9-teaser--news_all" data-component="m9-teaser" data-original-f-s="16px" style="font-size: 16px;">

DEWA supports Dubai Autism Center’s 20th Autism Awareness Campaign as Sustainability Partner

30 March 2026

                    <p class="teaser__copy" data-vl="As a part of its social responsibility, Dubai Electricity and Water Authority (DEWA) is supporting the Dubai Autism Centre’s 20th Autism Awareness Campaign as Sustainability Partner. The annual campaign is held under the patronage of HH Sheikh Hamdan bin Mohammed bin Rashid Al Maktoum, Crown Prince of Dubai, Deputy Prime Minister and Minister of Defence of the UAE, and Chairman of The Executive Council of Dubai. It aims to move beyond the concept of ‘formal accessibility’ to


Now I have the exact selectors. Let me also check the pagination and podcast page structure:


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Look for load more / pagination
for keyword in ['load-more', 'loadmore', 'pagination', 'next', 'Load More', 'Show More']:
    idx = html.lower().find(keyword.lower())
    if idx >= 0:
        print(f'Found \"{keyword}\" at {idx}:')
        print(html[max(0,idx-100):idx+200])
        print('---')
"

Find pagination/load more button in debug HTML


Found "next" at 232314:
arch--dm_item">
Show More

';
        var resultsPage = resultsUrl + '?term=' + searchTerm + '&page=1&category='+category;

        jQuery(ul + " li").not(":fi
---
Found "Show More" at 232358:
 class="m31-search--dm_link button button--text button--next">
Show More

';
        var resultsPage = resultsUrl + '?term=' + searchTerm + '&page=1&category='+category;

        jQuery(ul + " li").not(":first").remove();
        if (arr.length === 0
---


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Look for actual news pagination - year/month based navigation
for keyword in ['year', 'month', 'filter', 'category', 'page=', 'pager', 'api/news', 'api/articles']:
    idxs = [m.start() for m in re.finditer(keyword, html, re.IGNORECASE)][:2]
    for idx in idxs:
        snippet = html[max(0,idx-80):idx+200]
        if 'latest-news' in snippet or 'press' in snippet or 'news' in snippet.lower():
            print(f'Keyword \"{keyword}\" context:')
            print(snippet[:300])
            print('---')
            break
" 2>&1 | head -80

Look for news pagination and year/month filter patterns


(Bash completed with no output)


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Check for year-based URL patterns
years = re.findall(r'latest-news/(\d{4})', html)
print('Years in URLs:', sorted(set(years)))

# Count total teaser items
teasers = html.count('m9-teaser--news_all')
print('Total teaser items on page:', teasers)

# Look for API endpoint
api_matches = re.findall(r'api[^\"\'<>]{0,80}', html)
for a in api_matches[:10]:
    print('API:', a)
"

Check year patterns and API endpoints


Years in URLs: ['2026']
Total teaser items on page: 8
API: api.js?hl=en
API: api&auid=2003231174.1775276757&uaa=x86&uab=64&uafvl=Chromium%3B145.
API: api/sitecore/Search/SearchSuggestion
API: api/sitecore/Renderings/ListingV1
API: apital, approved the Board of Directors’ proposal to distribute cash dividends of A
API: apital) to shareholders for second half of the year 2025.
API: apital, approved the Board of Directors’ proposal to distribute cash dividends of
API: apital) to shareholders for second half of the year 2025.

API: api/sitecore/HappinessIndicator/PostData?type=Application&language=en&isloc
API: api/imgs/deaficon.png


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Look at the ListingV1 API call - that's likely the news listing API
idx = html.find('ListingV1')
print(html[max(0,idx-200):idx+500])
"

Inspect the ListingV1 API endpoint usage


<form action="/api/sitecore/Renderings/ListingV1" id="listing-form" method="post" data-original-f-s="16px" style="font-size: 16px;">                    <input type="hidden" name="itemId" value="d4a4a0bb-5c23-4b19-ad15-e2ff9c86a9ff" data-original-f-s="16px" style="font-size: 16px;">

        <div class="m35-filter" data-original-f-s="16px" style


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Get full listing form structure
idx = html.find('listing-form')
print(html[idx:idx+2000])
"

Inspect full listing form with filters and pagination


listing-form" method="post" data-original-f-s="16px" style="font-size: 16px;">                    <input type="hidden" name="itemId" value="d4a4a0bb-5c23-4b19-ad15-e2ff9c86a9ff" data-original-f-s="16px" style="font-size: 16px;">

                Show

                        The monthly period for the bill.

                            <select class="form-field__input form-field__input--select form-field__input--select-full form-filter" id="form-field-months" name="month" aria-describedby="description-for-months" data-parsley-errors-container="#description-for-months" title="All Months" data-original-f-s="16px" style="font-size: 16px;">
                                <option value="" selected="" data-original-f-s="16px" style="font-size: 16px;">All Months</option>
                                    <option value="1" data-original-f-s="16px" style="font-size: 16px;"


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Find year options and pagination structure
idx = html.find('form-field-months')
print(html[idx:idx+3000])
" 2>&1 | head -100

Find year/month filter options and pagination in the form


form-field-months" class="form-field__label" data-original-f-s="16px" style="font-size: 16px;">
                        The monthly period for the bill.

                            <select class="form-field__input form-field__input--select form-field__input--select-full form-filter" id="form-field-months" name="month" aria-describedby="description-for-months" data-parsley-errors-container="#description-for-months" title="All Months" data-original-f-s="16px" style="font-size: 16px;">
                                <option value="" selected="" data-original-f-s="16px" style="font-size: 16px;">All Months</option>
                                    <option value="1" data-original-f-s="16px" style="font-size: 16px;">January</option>
                                    <option value="2" data-original-f-s="16px" style="font-size: 16px;">February</option>
                                    <option value="3" selected="selected" data-original-f-s="16px" style="font-size: 16px;">March</option>
                                    <option value="4" data-original-f-s="16px" style="font-size: 16px;">April</option>
                                    <option value="5" data-original-f-s="16px" style="font-size: 16px;">May</option>
                                    <option value="6" data-original-f-s="16px" style="font-size: 16px;">June</option>
                                    <option value="7" data-original-f-s="16px" style="font-size: 16px;">July</option>
                                    <option value="8" data-original-f-s="16px" style="font-size: 16px;">August</option>
                                    <option value="9" data-original-f-s="16px" style="font-size: 16px;">September</option>
                                    <option value="10" data-original-f-s="16px" style="font-size: 16px;">October</option>
                                    <option value="11" data-original-f-s="16px" style="font-size: 16px;">November</option>
                                    <option value="12" data-original-f-s="16px" style="font-size: 16px;">December</option>
                            </select>


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Find year dropdown options
idx = html.find('form-field-years')
print(html[idx:idx+2000])
"

Extract year filter options


form-field-years" class="form-field__label" data-original-f-s="16px" style="font-size: 16px;">
                        years

                            <select class="form-field__input form-field__input--select form-field__input--select-full form-filter" id="form-field-years" name="year" aria-describedby="description-for-years" data-parsley-errors-container="#description-for-" title="All Years" data-original-f-s="16px" style="font-size: 16px;">
                                <option value="" selected="" data-original-f-s="16px" style="font-size: 16px;">All Years</option>
                                        <option value="2026" selected="" data-original-f-s="16px" style="font-size: 16px;">2026</option>
                                        <option value="2025" data-original-f-s="16px" style="font-size: 16px;">2025</option>
                                        <option value="2024" data-original-f-s="16px" style="font-size: 16px;">2024</option>
                                        <option value="2023" data-original-f-s="16px" style="font-size: 16px;">2023</option>
                                        <option value="2022" data-original-f-s="16px" style="font-size: 16px;">2022</option>
                                        <option value="2021" data-original-f-s="16px" style="font-size: 16px;">2021</option>
                                        <option value="2020" data-original-f-s="16px" style="font-size: 16px;">2020</option>
                                        <option value="2019" data-original-f-s="16px" style="font-size: 16px;">2019</option>


python3 -c "
from pathlib import Path
import re
html = Path('debug_podcasts.html').read_text()

# Find podcast items
print('File size:', len(html))
# Look for podcast/audio related patterns
for keyword in ['podcast', 'audio', 'mp3', 'teaser', 'listen']:
    count = html.lower().count(keyword.lower())
    if count:
        idx = html.lower().find(keyword.lower())
        print(f'\"{keyword}\" found {count} times, first at {idx}:')
        print(html[max(0,idx-100):idx+300])
        print('---')
"

Inspect podcast debug HTML for audio patterns


File size: 178215
"podcast" found 3 times, first at 137372:
22&u_w=1440&u_h=900&url=https%3A%2F%2Fwww.dewa.gov.ae%2Fabout-us%2Fmedia-publications%2Fpodcasts&frm=0&tiba=Not%20Found&hn=www.googleadservices.com&npa=0&pscdl=noapi&auid=2003231174.1775276757&uaa=x86&uab=64&uafvl=Chromium%3B145.0.7632.6%7CNot%253AA-Brand%3B99.0.0.0&uamb=0&uam=&uap=macOS&uapv=10_15_7&uaw=0&data=event%3Dgtag.co
---
"listen" found 69 times, first at 9496:
ITY:a,isNaN(b)?Number.NEGATIVE_INFINITY:b)};b.Am=r;b.Nh=e;b.Bq=function(a,b){e(a,b||1E4)};b.addEventListener=function(b,d,l,g){function n(){try{return l.apply(this,k(arguments))}catch(g){a.exception(g,"M1",d,b,g)}}void 0===g&&(g=!1);a.logInfo("M0",d,b);n.L=!0;b.addEventListener?b.addEventListener(d,n,g):b.attachEvent&&
b.attachEvent("on"+d,n)};b.loadScriptAsync=function(b){var g=document.createEle
---


python3 -c "
from pathlib import Path
import re
html = Path('debug_podcasts.html').read_text()

# The podcast page returned 404/Not Found - check title
title_match = re.search(r'<title[^>]*>([^<]+)</title>', html)
print('Page title:', title_match.group(1) if title_match else 'not found')

# Try finding the correct podcast URL
links = re.findall(r'href=\"([^\"]*podcast[^\"]*|[^\"]*audio[^\"]*|[^\"]*radio[^\"]*|[^\"]*sound[^\"]*|[^\"]*episode[^\"]*)[^\"]*\"', html, re.IGNORECASE)
print('Podcast-related links:', links[:20])
"

Check podcast page title and links


Page title: Not Found
Podcast-related links: ['/ar-AE/404?item=/about-us/media-publications/podcasts&user=extranet\\Anonymous&site=DEWA']


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Look for podcast links in the news page
links = re.findall(r'href=\"([^\"]*podcast[^\"]*|[^\"]*media-publications[^\"]+|[^\"]*audio[^\"]*|[^\"]*radio[^\"]*)[^\"]*\"', html, re.IGNORECASE)
print('Media/podcast links found:')
for l in sorted(set(links))[:20]:
    print(' ', l)

# Also check the nav/menu for correct podcast URL
nav = re.findall(r'media-publications[^\"]+', html)
print('\nAll media-publications paths:')
for n in sorted(set(nav))[:20]:
    print(' ', n)
"

Find correct podcast URL from navigation


Media/podcast links found:
  /ar-AE/about-us/media-publications/latest-news
  /en/about-us/media-publications/latest-news
  /en/about-us/media-publications/latest-news/2026/3/dewa-contributes-aed20-million
  /en/about-us/media-publications/latest-news/2026/3/dewa-promotes-the-noble-values
  /en/about-us/media-publications/latest-news/2026/3/dewa-supports-dubai-autism
  /en/about-us/media-publications/latest-news/2026/3/dewas-damj-initiative-supports-empowerment
  /en/about-us/media-publications/latest-news/2026/3/dewas-ownek-platform-streamlines
  /en/about-us/media-publications/latest-news/2026/3/empower-annual-general-meeting
  /en/about-us/media-publications/latest-news/2026/3/quote-by-he-saeed-mohammed
  /en/about-us/media-publications/latest-news/2026/3/quote-by-he-saeed-mohammed-al-tayer
  /en/about-us/media-publications/photo-gallery
  /en/about-us/media-publications/press-pack
  /en/about-us/media-publications/video-album-gallery
  https://www.dewa.gov.ae/ar-AE/about-us/media-publications/latest-news
  https://www.dewa.gov.ae/en/about-us/media-publications/latest-news

All media-publications paths:
  media-publications%2Flatest-news&frm=0&tiba=Latest%20News&hn=www.googleadservices.com&npa=0&pscdl=noapi&auid=2003231174.1775276757&uaa=x86&uab=64&uafvl=Chromium%3B145.0.7632.6%7CNot%253AA-Brand%3B99.0.0.0&uamb=0&uam=&uap=macOS&uapv=10_15_7&uaw=0&data=event%3Dgtag.config&rfmt=3&fmt=4
  media-publications/latest-news
  media-publications/latest-news/2026/3/dewa-contributes-aed20-million
  media-publications/latest-news/2026/3/dewa-promotes-the-noble-values
  media-publications/latest-news/2026/3/dewa-supports-dubai-autism
  media-publications/latest-news/2026/3/dewas-damj-initiative-supports-empowerment
  media-publications/latest-news/2026/3/dewas-ownek-platform-streamlines
  media-publications/latest-news/2026/3/empower-annual-general-meeting
  media-publications/latest-news/2026/3/quote-by-he-saeed-mohammed
  media-publications/latest-news/2026/3/quote-by-he-saeed-mohammed-al-tayer
  media-publications/photo-gallery
  media-publications/press-pack
  media-publications/video-album-gallery


python3 -c "
from pathlib import Path
import re
html = Path('debug_news.html').read_text()

# Look for RSS feed links
rss = re.findall(r'(?:rss|feed|atom)[^\"\'<>]{0,100}', html, re.IGNORECASE)
for r in rss[:15]:
    print(r)
"

Find RSS feed links in DEWA news page


source venv/bin/activate && python3 -c "
from playwright.sync_api import sync_playwright

with sync_playwright() as pw:
    browser = pw.chromium.launch(headless=True)
    ctx = browser.new_context(user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36')
    page = ctx.new_page()

    # Visit a single PR page to see if podcast is embedded
    page.goto('https://www.dewa.gov.ae/en/about-us/media-publications/latest-news/2026/3/dewa-supports-dubai-autism', wait_until='networkidle', timeout=30000)
    page.wait_for_timeout(3000)

    from pathlib import Path
    Path('debug_pr_detail.html').write_text(page.content())
    print('Saved PR detail page')
    browser.close()
" 2>&1

Fetch a single PR detail page to check podcast structure


Saved PR detail page


python3 -c "
from pathlib import Path
import re
html = Path('debug_pr_detail.html').read_text()

print('Page size:', len(html))

# Look for audio/podcast
for kw in ['audio', 'podcast', 'mp3', 'soundcloud', 'spotify', 'anchor', 'buzzsprout', 'listen', 'player']:
    count = html.lower().count(kw)
    if count:
        idx = html.lower().find(kw)
        print(f'\"{kw}\" ({count}x):')
        print(html[max(0,idx-100):idx+300])
        print('---')
"

Check if PR detail pages contain podcast/audio content


Page size: 279130
"podcast" (4x):
div class="grid__row" data-original-f-s="16px" style="font-size: 16px;">

            <iframe src="https://player.rss.com/dewa-new
---
"listen" (2x):
n(){
			showSuggestion();

			//},300);

        }

        return true;
	}

    inputBox.addEventListener("keypress", function (event) {

            if (event.key === "Enter") {
                 $('.m31-search--input').val(inputBox.value.replaceAll(/\u200f/g, '').replaceAll('@', '').replaceAll('*', '').replaceAll('{', '').replaceAll('}', '').replaceAll('[', '').replaceAll(']', '').replaceAll
---
"player" (2x):
rapper rs_skip" data-original-f-s="16px" style="font-size: 16px;">
            <iframe src="https://player.rss.com/dewa-news/2679905?theme=light&share=false&about=false&hl=aGlkZV9sb2dv" title="DEWA supports Dubai Autism Center’s 20th Autism Awareness Campaign as Sustainability Partner" width="100%" height="100%" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted
---


Excellent! Podcasts are embedded as `rss.com` iframes inside each PR. Let me get the full iframe structure and check the RSS feed:


python3 -c "
from pathlib import Path
import re
html = Path('debug_pr_detail.html').read_text()

# Get full podcast iframe
idx = html.find('player.rss.com')
print(html[max(0,idx-300):idx+500])
"

Get full podcast iframe HTML


DEWA su


source venv/bin/activate && python3 -c "
from playwright.sync_api import sync_playwright
from pathlib import Path

with sync_playwright() as pw:
    browser = pw.chromium.launch(headless=True)
    ctx = browser.new_context(user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36')
    page = ctx.new_page()

    # Fetch the RSS.com podcast feed for DEWA
    page.goto('https://rss.com/podcasts/dewa-news/', wait_until='networkidle', timeout=30000)
    page.wait_for_timeout(3000)
    Path('debug_rss_feed.html').write_text(page.content())
    print('Title:', page.title())
    print('Saved rss.com page')
    browser.close()
" 2>&1

Fetch DEWA RSS.com podcast feed page


Title: DEWA Podcasts | Podcast on RSS.com
Saved rss.com page


python3 -c "
from pathlib import Path
import re
html = Path('debug_rss_feed.html').read_text()

# Find episode items
for kw in ['episode', 'feed', 'mp3', 'dewa-news/', 'duration']:
    count = html.lower().count(kw)
    if count > 0:
        idx = html.lower().find(kw)
        print(f'\"{kw}\" ({count}x) first occurrence:')
        print(html[max(0,idx-100):idx+300])
        print('---')
"

Inspect RSS.com podcast page structure


"episode" (639x) first occurrence:
ion:all .2s ease;border:1px solid #595959}.custom-website-btn:hover{color:#fff;border-color:#fff}

.episode-wrp__share-elements[data-v-671468fe]{display:flex;align-items:center}.episode-wrp__share-elements .share-btn[data-v-671468fe]{display:flex;justify-content:center}.share-btn[data-v-671468fe]{display:flex;width:100%;padding:7px 24px;border-radius:4px;align-items:center;text-decoration:none;box
---
"feed" (104x) first occurrence:
="alternate" type="application/rss+xml" title="DEWA Podcasts" href="https://media.rss.com/dewa-news/feed.xml"><script src="//bat.bing.com/bat.js" async=""></script><script async="" src="https://bat.bing.com/p/insights/s/0.8.59"></script><script async="" src="https://bat.bing.com/p/insights/t/17227567"></script><script src="//bat.bing.com/bat.js" async=""></script><script type="text/javascript" asy
---
"mp3" (20x) first occurrence:
419bd2f8db44aea2e759e6.png",episode_asset:"2026_04_01_08_57_46_9e4716e2-4221-4269-9b02-a9f6cb3705f8.mp3",episode_keywords:[{id:k,label:l},{id:m,label:d},{id:n,label:o}],locations:[],transcript_url:"https:\u002F\u002Ftranscripts.rss.com\u002F331008\u002F2679905\u002Ftranscript",ai_content:c,episode_asset_url:"https:\u002F\u002Fcontent.rss.com\u002Fepisodes\u002F331008\u002F2679905\u002Fdewa-news\u0
---
"dewa-news/" (70x) first occurrence:
S.com"><meta data-n-head="ssr" data-hid="og:image" property="og:image" content="https://img.rss.com/dewa-news/900/20250523_040515_a31ff5ab754aa1346615c1a67cc806b6.jpg"><meta data-n-head="ssr" data-hid="og:image:secure_url" property="og:image:secure_url" content="https://img.rss.com/dewa-news/900/20250523_040515_a31ff5ab754aa1346615c1a67cc806b6.jpg"><meta data-n-head="ssr" data-hid="og:description"
---
"duration" (62x) first occurrence:
B 30%, #ededed 30%)}.progress:indeterminate,.progress-wrapper.is-not-native:indeterminate{animation-duration:1.5s;animation-iteration-count:infinite;animation-name:moveIndeterminate;animation-timing-function:linear;background-color:#ededed;background-image:linear-gradient(to right, #4a4a4a 30%, #ededed 30%);background-position:top left;background-repeat:no-repeat;background-size:150% 150%}.progres
---


There's a direct RSS feed at `https://media.rss.com/dewa-news/feed.xml`. This is perfect — no browser needed for podcasts. Let me fetch it:


curl -s "https://media.rss.com/dewa-news/feed.xml" | head -100

Fetch DEWA podcast RSS feed XML