Simple Web Scraper
What you'll build
A Python script that fetches a URL, extracts all hyperlinks using regex, and prints them — a mini web scraper using only the standard library.
Goals
- Fetch web content with urllib.request
- Extract links with regular expressions
- Handle network errors with try/except
- Clean and deduplicate extracted URLs
Prerequisites
- Strings and string methods
- Regular expressions (re module)
- Exception handling
Steps
- 1
Fetch a web page
Import urllib.request and define fetch_page(url) that returns the HTML content as a string. Handle network errors with try/except.
Codeimport urllib.request def fetch_page(url): try: with urllib.request.urlopen(url) as response: html = response.read().decode('utf-8') return html except Exception as e: print(f"Error fetching {url}: {e}") return NoneExpected result
fetch_page('https://example.com') returns the HTML content as a string.
- 2
Extract links with regex
Import re and define extract_links(html) that finds all href attributes in anchor tags. Return a deduplicated, sorted list.
Codeimport re def extract_links(html): pattern = r'href=["\']([^"\']+)["\']' links = re.findall(pattern, html) return sorted(set(links))Expected result
extract_links(html) returns a sorted list of unique URLs found in href attributes.
- 3
Put it together
Define scrape(url) that fetches the page, extracts links, and prints each one with a number. Test with a simple URL.
Codedef scrape(url): html = fetch_page(url) if html is None: return links = extract_links(html) print(f"Found {len(links)} unique links:") for i, link in enumerate(links, 1): print(f" {i}. {link}") # scrape('https://example.com') # Uncomment to testExpected result
A numbered list of unique links found on the page.
- 4
Add input validation & exception handling
Add a try/except block in the main function to handle user input errors, using ValueError to catch illegal conversions. Python's exception handling lets the program degrade gracefully on bad input instead of crashing.
Codedef safe_input(prompt, validator=None): """Read input with validation and error handling.""" while True: try: raw = input(prompt) value = float(raw) if validator and not validator(value): raise ValueError(f"Value {value} failed validation") return value except ValueError as err: print(f"Invalid input: {err}. Please try again.") # Usage age = safe_input("Enter your age: ", lambda x: x > 0)Expected result
Invalid input (non-number or failed validation) prompts for re-entry; valid input returns the number.
- 5
Write unit tests
Write tests for the core function using unittest or assert statements, covering normal paths, boundary values, and exceptions. Tests are the safety net for code quality.
Codeimport unittest class TestCoreFunc(unittest.TestCase): def test_normal(self): self.assertEqual(core_func(5), expected_5) def test_boundary(self): self.assertEqual(core_func(0), expected_0) def test_exception(self): with self.assertRaises(ValueError): core_func(-1) if __name__ == "__main__": unittest.main()Expected result
Running python test.py prints OK with all three test cases passing.
Try it yourself
import urllib.request
import re
# TODO: define fetch_page, extract_links, and scrape
# Then test with: scrape('https://example.com')
import urllib.request
import re
def fetch_page(url):
try:
with urllib.request.urlopen(url) as response:
html = response.read().decode('utf-8')
return html
except Exception as e:
print(f"Error fetching {url}: {e}")
return None
def extract_links(html):
pattern = r'href=["\']([^"\']+)["\']'
links = re.findall(pattern, html)
return sorted(set(links))
def scrape(url):
html = fetch_page(url)
if html is None:
return
links = extract_links(html)
print(f"Found {len(links)} unique links:")
for i, link in enumerate(links, 1):
print(f" {i}. {link}")
scrape('https://example.com')What you learned
You fetched web content, parsed HTML with regex, handled network errors, and deduplicated results — the core of web scraping, using only Python's standard library.
Related Resources
Related Lessons
- Regex
Pattern matching for links.
- Exceptions
Network error handling.
Related Practice
- Regex Module Quiz
Practice re module.
- Predict try/except
Exception handling.
Related Cheatsheets
- Python Cheatsheet
urllib and re reference.
Related Errors
- URL Error
Invalid URLs.
- Unicode Decode Error
Encoding issues.
Related Glossary
- Python Module
Standard library modules.
- Exception
Error handling.
Related Glossary
A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.
- Python Module
Standard library modules.
- Exception
Error handling.
Related Errors
Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.
- URL Error
Invalid URLs.
- Unicode Decode Error
Encoding issues.