Generators
Create lazy, memory-efficient sequences with generator functions, yield, and generator expressions.
What you'll learn
- Write generator functions with yield
- Understand lazy evaluation and memory efficiency
- Use generator expressions for inline generators
- Chain generators in pipelines
- Handle infinite generators with itertools
Concept
What Is a Generator?
A generator is a function that produces a sequence of values lazily — one at a time, on demand — instead of computing them all upfront and storing them in memory. You create one with the yield keyword:
def count_up_to(n):
i = 0
while i < n:
yield i
i += 1
# Create the generator — nothing runs yet
counter = count_up_to(5)
# Values are produced one at a time
print(next(counter)) # 0
print(next(counter)) # 1
print(next(counter)) # 2
When you call a generator function, it returns a generator object without running any code. Each call to next() runs the function until it hits yield, produces that value, and pauses. The next next() resumes from where it left off. When the function returns (or falls off the end), StopIteration is raised.
yield vs return
return— returns a value and ends the functionyield— produces a value and pauses the function, to be resumed later
def squares(n):
for x in range(n):
yield x ** 2
for sq in squares(5):
print(sq) # 0, 1, 4, 9, 16
A generator function can have multiple yield statements. Each yield produces a value and suspends execution. The function resumes from the yield on the next next() call.
Why Generators? Memory Efficiency
The key advantage is memory. Compare:
# List — all 1 million values in memory at once
squares_list = [x ** 2 for x in range(1_000_000)]
# Uses ~8 MB of memory
# Generator — only one value in memory at a time
squares_gen = (x ** 2 for x in range(1_000_000))
# Uses ~100 bytes of memory
For large or infinite sequences, generators are essential. You can process a 10 GB file line by line with a generator using only kilobytes of memory.
Generator Expressions
The parenthesized version of a list comprehension creates a generator:
# List comprehension — eager, all values in memory
squares_list = [x ** 2 for x in range(10)]
# Generator expression — lazy, one value at a time
squares_gen = (x ** 2 for x in range(10))
print(sum(squares_gen)) # 285 — computed without storing all values
Generator expressions are perfect for feeding into sum(), max(), min(), or any function that consumes an iterable. They avoid building an intermediate list.
Sending Values with .send()
Generators can receive values from the caller using .send():
def accumulator():
total = 0
while True:
value = yield total
if value is None:
break
total += value
acc = accumulator()
next(acc) # prime the generator — runs to first yield
print(acc.send(10)) # 10
print(acc.send(20)) # 30
print(acc.send(5)) # 35
This makes generators coroutines — they can both produce and consume values. This is an advanced feature used in async programming.
Chaining Generators
Generators compose beautifully — you can pipe them together:
def numbers():
for n in range(1, 20):
yield n
def squared(seq):
for n in seq:
yield n ** 2
def evens(seq):
for n in seq:
if n % 2 == 0:
yield n
# Pipeline: numbers → squared → evens
pipeline = evens(squared(numbers()))
print(list(pipeline)) # [4, 16, 36, 64, 100, 144, 196, 256, 324]
Each stage processes values lazily — no intermediate lists are created. This is the foundation of streaming data pipelines.
Infinite Generators
Generators can be infinite — they never run out of values:
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
fib = fibonacci()
for _ in range(10):
print(next(fib)) # 0, 1, 1, 2, 3, 5, 8, 13, 21, 34
An infinite generator is safe as long as you only consume a finite number of values (with next(), itertools.islice(), or a bounded loop). itertools provides many useful generators:
from itertools import islice, count, cycle
# count() — infinite counter: 0, 1, 2, 3, ...
for n in islice(count(), 5):
print(n) # 0, 1, 2, 3, 4
# cycle() — repeat a sequence forever
for item in islice(cycle(["A", "B", "C"]), 7):
print(item) # A, B, C, A, B, C, A
yield from — Delegating to Sub-generators
yield from delegates to another generator, flattening nested generators:
def inner():
yield 1
yield 2
def outer():
yield 0
yield from inner() # yields all values from inner()
yield 3
print(list(outer())) # [0, 1, 2, 3]
This is cleaner than manually iterating and yielding each value from the sub-generator.
Example
# Basic generator with yield
def count_up_to(n):
i = 0
while i < n:
yield i
i += 1
print(list(count_up_to(5))) # [0, 1, 2, 3, 4]
# Generator vs list — memory efficiency
import sys
big_list = [x ** 2 for x in range(10000)]
big_gen = (x ** 2 for x in range(10000))
print(f"List size: {sys.getsizeof(big_list)} bytes")
print(f"Gen size: {sys.getsizeof(big_gen)} bytes")
# Using next() manually
counter = count_up_to(3)
print(next(counter)) # 0
print(next(counter)) # 1
print(next(counter)) # 2
# Infinite generator — Fibonacci
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
fib = fibonacci()
fib_10 = [next(fib) for _ in range(10)]
print(f"Fibonacci: {fib_10}")
# Chaining generators in a pipeline
def numbers():
for n in range(1, 10):
yield n
def squared(seq):
for n in seq:
yield n ** 2
def evens(seq):
for n in seq:
if n % 2 == 0:
yield n
pipeline = evens(squared(numbers()))
print(f"Pipeline: {list(pipeline)}")
# yield from — delegating
def inner():
yield 1
yield 2
def outer():
yield 0
yield from inner()
yield 3
print(f"yield from: {list(outer())}") # [0, 1, 2, 3]
# Reading a large file lazily
def read_lines(filename):
with open(filename) as f:
for line in f:
yield line.strip() # one line at a time, low memory
Shows a basic generator, memory comparison vs list, manual next() usage, an infinite Fibonacci generator, a chained pipeline (numbers → squared → evens), yield from delegation, and a lazy file reader pattern.
Try it
- Text Counter
Generators are perfect for streaming text processing line by line.
- JSON Formatter
Generators can stream large JSON arrays without loading everything into memory.
Common mistakes
The mistake
Trying to reuse a generator that has been exhausted
The fix
Generators are single-use — once consumed, they are empty and next() raises StopIteration. To reuse, recreate the generator: gen = my_generator() again, or convert to a list first if you need to iterate multiple times.
The mistake
Creating an infinite generator and iterating with a for loop without a break
The fix
An infinite generator (while True: yield) never stops. Always use a bounded consumer: next() a fixed number of times, itertools.islice(gen, n), or a for loop with a break condition. A bare 'for x in infinite_gen:' hangs forever.
Related Resources
Related Cheatsheets
- Python Cheatsheet
yield, generator expressions, send(), and itertools reference.
Practice Generators
4 exercises
Look up unfamiliar terms
A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.
Build real projects
Apply what you learned by building guided projects with starter code and solutions.
Decode error messages
Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.