Skip to content

Generators and Iterators

Medium ~30 min

What you'll build

A set of generators: fibonacci (infinite), chunker (split iterable into chunks), and a pipeline that filters/transforms a large data stream lazily.

Goals

  • Use yield to produce values lazily
  • Write an infinite generator
  • Chain generators into a pipeline
  • Compare memory usage vs list-based approaches

Prerequisites

  • for loops and iterables
  • Functions
  • List comprehensions

Steps

  1. 1

    Write a Fibonacci generator

    A generator uses yield to produce values one at a time, pausing between each. Write an infinite Fibonacci generator — it never returns, but yields forever.

    Code
    def fibonacci():
        a, b = 0, 1
        while True:
            yield a
            a, b = b, a + b
    
    # Take the first 10 values
    fib = fibonacci()
    print([next(fib) for _ in range(10)])
    # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

    Expected result

    First 10 Fibonacci numbers: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34].

  2. 2

    Use itertools.islice

    Since the generator is infinite, use itertools.islice to take a finite number of values without list slicing (which would hang forever).

    Code
    from itertools import islice
    
    print(list(islice(fibonacci(), 15)))
    # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377]

    Expected result

    First 15 Fibonacci numbers as a list.

  3. 3

    Write a chunker generator

    Write a generator that yields chunks of size n from an iterable. This is useful for batch processing without loading everything into memory.

    Code
    def chunker(iterable, size):
        chunk = []
        for item in iterable:
            chunk.append(item)
            if len(chunk) == size:
                yield chunk
                chunk = []
        if chunk:
            yield chunk  # yield the remaining partial chunk
    
    for chunk in chunker(range(10), 3):
        print(chunk)
    # [0, 1, 2], [3, 4, 5], [6, 7, 8], [9]

    Expected result

    Four chunks: three of size 3 and one of size 1.

  4. 4

    Build a processing pipeline

    Chain generators: read numbers, filter evens, square them, take first 5. Each step is a generator, so the whole pipeline is lazy — nothing runs until you consume the output.

    Code
    def numbers():
        n = 0
        while True:
            yield n
            n += 1
    
    def evens(src):
        for n in src:
            if n % 2 == 0:
                yield n
    
    def squared(src):
        for n in src:
            yield n * n
    
    pipeline = squared(evens(numbers()))
    print(list(islice(pipeline, 5)))
    # [0, 4, 16, 36, 64]

    Expected result

    First 5 squares of even numbers: [0, 4, 16, 36, 64].

  5. 5

    Compare memory with lists

    Show that a generator processing 10M numbers uses constant memory, while a list would use ~80MB. Use sys.getsizeof to compare.

    Code
    import sys
    
    # Generator: ~128 bytes regardless of how many it could produce
    gen = (x**2 for x in range(10_000_000))
    print(f"Generator size: {sys.getsizeof(gen)} bytes")
    
    # List: would be ~80MB for 10M integers
    # lst = [x**2 for x in range(10_000_000)]  # don't actually run this
    print("A list of 10M ints would be ~80MB")

    Expected result

    Generator is ~128 bytes; a list would be megabytes.

Try it yourself

What you learned

You mastered generators — Python's way to process large or infinite data streams with constant memory. They power file iteration, streaming, and async pipelines.

Related Resources

Related Lessons

  • Loops

    Generators are iterables used in for loops.

Look up unfamiliar terms

A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.

View glossary

Decode error messages

Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.

View errors

← Back to topic