Python Generators and Iterators
Learn how Python yield, generator functions, generator expressions, and iterator exhaustion support lazy data processing.
Python generators produce values one at a time, which can make data pipelines clearer and avoid building an entire result list in memory. This guide explains iterables, iterators, yield, generator expressions, exhaustion, and practical limits with examples you can adapt.
Iterable, iterator, and generator: how they relate
An iterable is an object Python can obtain an iterator from, commonly with iter(value). Lists, strings, dictionaries, and files are familiar examples. An iterator produces the next value when asked; it implements __next__() and raises StopIteration when exhausted. A generator is a convenient kind of iterator created by a generator function or generator expression.
A list stores its values for reuse. An iterator is usually consumed as it is read. That difference matters when you need to loop twice, index values, or keep the results around.
Use yield to produce a stream
A function containing yield is a generator function. Calling it creates a generator object; the body begins running when iteration asks for its first value. At each yield, execution pauses while local state is retained. The next request resumes from that point.
def even_numbers(limit):
for number in range(limit + 1):
if number % 2 == 0:
yield number
values = even_numbers(6)
print(next(values)) # 0
print(next(values)) # 2
for value in values:
print(value) # 4, then 6The loop and calls to next() share the same iterator. After the two explicit calls, the loop continues at the next value rather than starting over. A later attempt to read an exhausted generator raises StopIteration; a for loop handles that signal automatically.
Use return when a function should finish with one result. In a generator, return ends iteration; it does not emit another item for a normal for loop.
Transform data without an intermediate list
Generators are useful when values can be processed one by one. For example, filter a large stream of log lines and parse only matching records:
def error_lines(file_object):
for line in file_object:
if "ERROR" in line:
yield line.rstrip()
with open("application.log", encoding="utf-8") as log_file:
for line in error_lines(log_file):
print(line)The generator does not first create a second list containing every matching line. It requests each line as the loop needs it. The surrounding file context manager keeps the file open while the generator is consumed and closes it afterward. If a generator owns a resource itself, make cleanup explicit with a context manager or try/finally, and ensure callers consume or close it appropriately.
Generator expressions for short pipelines
For a simple transformation, parentheses create a generator expression. Compare it with a list comprehension:
squares_list = [number * number for number in range(1_000_000)]
squares_stream = (number * number for number in range(1_000_000))
total = sum(squares_stream)The list comprehension constructs and stores all the square values. The generator expression computes each value as it is requested. This can reduce the memory needed for an intermediate result, but it does not make every operation constant-memory: a consumer such as sorted() must retain values to order them, and the generator's input or other pipeline stages may also store data.
Prefer a list when you need repeated iteration, random access, or a stable snapshot. Prefer a generator when you can consume values once, in order, and incrementally.
Chaining generators and built-in tools
A generator can feed another iterator-friendly operation. This keeps transformations readable without manually managing indexes:
def normalized_words(lines):
for line in lines:
for word in line.split():
yield word.casefold()
with open("notes.txt", encoding="utf-8") as source:
words = normalized_words(source)
long_words = (word for word in words if len(word) >= 8)
for word in long_words:
print(word)Python's itertools module provides building blocks such as islice for consuming a bounded portion of an iterator. Check whether each operation consumes its input, returns a new iterator, or buffers values when designing a large pipeline.
Common pitfalls
- Trying to reuse a consumed generator: create a fresh generator by calling its function again, or materialize values into a list when reuse is required.
- Assuming a generator runs when created: its body normally starts when the first value is requested, so exceptions inside it may occur during iteration.
- Calling
list()too early: this collects all remaining values and removes the streaming memory benefit. - Returning a generator from a closed resource: if the generator reads a file, keep that file open for the full iteration or have the generator manage its own resource scope.
- Expecting infinite input to finish: a generator may produce values forever; use a limit or a condition when the consumer needs a bounded result.
When a generator is the right tool
Use a generator when the output is naturally a sequence, each item can be produced on demand, and consumers can process it in order. It is a good fit for reading files, transforming records, generating ranges of derived values, and composing filters.
A generator is not automatically faster, and it does not make an expensive calculation cheap. It trades stored results for incremental production. Measure the real workload and choose the simplest structure that fits its memory, reuse, and resource-lifetime needs.
Further reading
- Python Functional Programming HOWTO: generators
- Python Language Reference: yield expressions
- Python itertools documentation
Want a structured Java learning path?
Explore Jaipur Engineers course options to connect Java fundamentals, debugging, backend development and project practice in a guided learning plan.