Python Automation Scripts for Web Developers

A lot of web work is repetitive: checking links, compressing images, pulling reports. Here are small Python scripts I use to automate it. Each one fits on one screen.

1. Broken link checker

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

def check_links(page_url):
    html = requests.get(page_url, timeout=15).text
    soup = BeautifulSoup(html, "html.parser")
    for a in soup.select("a[href]"):
        url = urljoin(page_url, a["href"])
        if not url.startswith("http"):
            continue
        try:
            r = requests.head(url, allow_redirects=True, timeout=10)
            if r.status_code >= 400:
                print(r.status_code, url)
        except requests.RequestException as e:
            print("ERR", url, e)

check_links("https://example.com")

2. Uptime check with an email alert

import smtplib, requests
from email.message import EmailMessage

SITES = ["https://example.com", "https://shop.example.com"]

for site in SITES:
    try:
        ok = requests.get(site, timeout=10).status_code < 400
    except requests.RequestException:
        ok = False
    if not ok:
        msg = EmailMessage()
        msg["Subject"] = f"DOWN: {site}"
        msg["From"] = msg["To"] = "[email protected]"
        msg.set_content(f"{site} did not respond correctly.")
        with smtplib.SMTP_SSL("smtp.example.com", 465) as s:
            s.login("[email protected]", "app-password")
            s.send_message(msg)

3. Bulk image optimisation to WebP

from pathlib import Path
from PIL import Image

for path in Path("uploads").rglob("*.[jp][pn]g"):
    img = Image.open(path)
    if img.width > 1600:
        img = img.resize((1600, int(img.height * 1600 / img.width)))
    img.save(path.with_suffix(".webp"), "WEBP", quality=80)
    print("optimised", path)

I used exactly this kind of script to shrink the screenshots on this portfolio from 1 to 2 MB PNGs to lightweight WebP files.

4. API → Google Sheet sync

import gspread, requests

orders = requests.get("https://api.example.com/orders?status=new",
                      headers={"Authorization": "Bearer TOKEN"}).json()

sheet = gspread.service_account("creds.json").open("Orders").sheet1
for o in orders:
    sheet.append_row([o["id"], o["customer"], o["total"], o["created_at"]])

5. Scheduling: cron is enough

# crontab -e
*/5 * * * *  /usr/bin/python3 /opt/scripts/uptime.py
0 3 * * *    /usr/bin/python3 /opt/scripts/optimise_images.py >> /var/log/img.log 2>&1

Use a virtualenv and full paths in cron. Most "it works in my terminal but not in cron" problems come from the environment.

Tips for scripts that keep working

• Keep secrets in environment variables, not in the script.
• Always set request timeouts; a hung request will block a cron job forever.
• Log to a file, and alert on failure, not just on success.
• Respect robots.txt and rate limits when scraping.

Need a workflow automated for your business? Tell me about it.