Skip to content

Simple Web Scraper

Medium ~25 min

What you'll build

A Python script that fetches a URL, extracts all hyperlinks using regex, and prints them — a mini web scraper using only the standard library.

Goals

  • Fetch web content with urllib.request
  • Extract links with regular expressions
  • Handle network errors with try/except
  • Clean and deduplicate extracted URLs

Prerequisites

  • Strings and string methods
  • Regular expressions (re module)
  • Exception handling

Steps

  1. 1

    Fetch a web page

    Import urllib.request and define fetch_page(url) that returns the HTML content as a string. Handle network errors with try/except.

    Code
    import urllib.request
    
    def fetch_page(url):
        try:
            with urllib.request.urlopen(url) as response:
                html = response.read().decode('utf-8')
            return html
        except Exception as e:
            print(f"Error fetching {url}: {e}")
            return None

    Expected result

    fetch_page('https://example.com') returns the HTML content as a string.

  2. 2

    Extract links with regex

    Import re and define extract_links(html) that finds all href attributes in anchor tags. Return a deduplicated, sorted list.

    Code
    import re
    
    def extract_links(html):
        pattern = r'href=["\']([^"\']+)["\']'
        links = re.findall(pattern, html)
        return sorted(set(links))

    Expected result

    extract_links(html) returns a sorted list of unique URLs found in href attributes.

  3. 3

    Put it together

    Define scrape(url) that fetches the page, extracts links, and prints each one with a number. Test with a simple URL.

    Code
    def scrape(url):
        html = fetch_page(url)
        if html is None:
            return
        links = extract_links(html)
        print(f"Found {len(links)} unique links:")
        for i, link in enumerate(links, 1):
            print(f"  {i}. {link}")
    
    # scrape('https://example.com')  # Uncomment to test

    Expected result

    A numbered list of unique links found on the page.

  4. 4

    Add input validation & exception handling

    Add a try/except block in the main function to handle user input errors, using ValueError to catch illegal conversions. Python's exception handling lets the program degrade gracefully on bad input instead of crashing.

    Code
    def safe_input(prompt, validator=None):
        """Read input with validation and error handling."""
        while True:
            try:
                raw = input(prompt)
                value = float(raw)
                if validator and not validator(value):
                    raise ValueError(f"Value {value} failed validation")
                return value
            except ValueError as err:
                print(f"Invalid input: {err}. Please try again.")
    
    # Usage
    age = safe_input("Enter your age: ", lambda x: x > 0)

    Expected result

    Invalid input (non-number or failed validation) prompts for re-entry; valid input returns the number.

  5. 5

    Write unit tests

    Write tests for the core function using unittest or assert statements, covering normal paths, boundary values, and exceptions. Tests are the safety net for code quality.

    Code
    import unittest
    
    class TestCoreFunc(unittest.TestCase):
        def test_normal(self):
            self.assertEqual(core_func(5), expected_5)
    
        def test_boundary(self):
            self.assertEqual(core_func(0), expected_0)
    
        def test_exception(self):
            with self.assertRaises(ValueError):
                core_func(-1)
    
    if __name__ == "__main__":
        unittest.main()

    Expected result

    Running python test.py prints OK with all three test cases passing.

Try it yourself

What you learned

You fetched web content, parsed HTML with regex, handled network errors, and deduplicated results — the core of web scraping, using only Python's standard library.

Related Resources

Related Lessons

Related Practice

Related Cheatsheets

Related Errors

Related Glossary

Related Glossary

A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.

Related Errors

Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.

← Back to topic