Python Generators: Using yield to Create Generators

Introduction to Python Generators

Generator functions allow us to declare a function that behaves like an iterator, i.e. it can be used in a for loop. Instead of returning a single value like a regular function, a generator function yields a series of values, one at a time, pausing the function's state between each yield. This makes generators particularly useful for handling large datasets or streams of data efficiently.

In this tutorial, we'll explore how to create and use generators in Python with examples and, explanations.

Example 1: Basic Generator Function

A generator function is defined like a regular function but uses the ‘yield’ statement instead of 'return'.

This example demonstrates how to create a simple generator that yields numbers from 1 to 5. The 'yield' statement allows the function to return a value and pause its state, making it efficient for iteration.

Code:

# A simple generator function that yields numbers from 1 to 5
def simple_generator():
    for i in range(1, 6):  # Iterate from 1 to 5
        yield i  # Yield the current value of i

# Using the generator
gen = simple_generator()

# Iterating through the generator
for value in gen:
    print(value)

Output:

1
2
3
4
5

Explanation:

• 'yield’ Statement:
• The 'yield' statement produces a value and pauses the function's execution, saving its state for the next time the generator is called.
• Using the Generator:
• The generator is used like an iterator. The 'for' loop automatically calls ‘next()’ on the generator, retrieving one value at a time.

Example 2: Generator for Fibonacci Sequence

Generators are well-suited for producing infinite or large sequences, such as the Fibonacci sequence.

This example shows how to create a generator for the Fibonacci sequence. The generator efficiently produces an infinite sequence, yielding one number at a time, without needing to store the entire sequence in memory.

Code:

# Generator expression to create a generator for squares of numbers
squares_gen = (x * x for x in range(1, 6))

# Iterating through the generator expression
for square in squares_gen:
    print(square)

Output:

0
1
4
9
16
25

Explanation:

• Generator Expression:
• The generator expression '(x * x for x in range(1, 6))' creates a generator that produces the squares of numbers from 1 to 5.
• Using the Generator:
• The 'for' loop iterates through the generator, retrieving each square and printing it.

Example 3: Controlling Generator Execution with send(), throw(), and close()

Generators in Python support additional methods like ‘send()’, ‘throw()’, and ‘close()’ to control their execution more precisely.

This example shows how to control a generator's execution using the 'send()', 'throw()', and 'close()' methods. The generator can accept values sent to it and perform cleanup when closed.

Code:

# Generator that accepts values via send() and stops on close()
def controlled_generator():
    try:
        while True:
            received = yield  # Yield None and wait for a value to be sent
            print(f"Received: {received}")
    except GeneratorExit:
        print("Generator closed")

# Using the generator
gen = controlled_generator()

# Start the generator
next(gen)  # Advance to the first yield

# Send values to the generator
gen.send("Hello")  # Output: Received: Hello
gen.send("World")  # Output: Received: World

# Closing the generator
gen.close()  # Output: Generator closed

Output:

Received: Hello
Received: World
Generator closed

Explanation:

• 'send()' Method:
• The 'send()' method resumes the generator and sends a value to it, which is received at the point of the last yield.
• 'throw()' Method:
• (Not used in this example, but available) This method allows you to throw an exception inside the generator.
• 'close()' Method:
• The 'close()' method stops the generator by raising a ‘GeneratorExit’ exception inside it, which can be caught to perform cleanup.

Example 4: Generator for Reading Large Files

Generators are ideal for reading large files line by line, allowing you to process files that don't fit into memory.

This example demonstrates how to use a generator to read a large file line by line. This method is memory-efficient and suitable for processing large files that might not fit into memory.

Code:

# Generator function to read a file line by line
def read_large_file(file_name):
    with open(file_name) as file:
        for line in file:
            yield line.strip()  # Yield each line without the newline character

# Using the generator to process the file
for line in read_large_file("data.txt"):
    print(line)

Output:

1.000000000000000000e+00 2.000000000000000000e+00 3.000000000000000000e+00
4.000000000000000000e+00 5.000000000000000000e+00 6.000000000000000000e+00
7.000000000000000000e+00 8.000000000000000000e+00 9.000000000000000000e+00

Explanation:

• File Reading Generator ('read_large_file'):
• The generator reads a file line by line, yielding each line as it is read.
• This approach is memory-efficient because it doesn't load the entire file into memory.

Example 5: Chaining Generators

You can chain multiple generators together to create a more complex sequence.

This example demonstrates how to chain multiple generators together using yield from. The combined generator produces a sequence by iterating through each chained generator in order.

Code:

# Generator for even numbers
def even_numbers():
    n = 2
    while True:
        yield n
        n += 2

# Generator for odd numbers
def odd_numbers():
    n = 1
    while True:
        yield n
        n += 2

# Generator that chains even and odd numbers
def even_odd_chain(even_gen, odd_gen):
    yield from even_gen  # Yield all values from the even number generator
    yield from odd_gen   # Then yield all values from the odd number generator

# Creating generators
even_gen = even_numbers()
odd_gen = odd_numbers()

# Chaining generators
chain_gen = even_odd_chain(even_gen, odd_gen)

# Printing the first 10 numbers from the chained generator
for _ in range(10):
    print(next(chain_gen))

Output:

2
4
6
8
10
12
14
16
18
20

Explanation:

• Chaining Generators:
• The 'even_odd_chain' generator uses 'yield from' to chain the even and odd number generators.
• It first yields all values from the 'even_gen' and then continues with the 'odd_gen'.

Note:

• The yield from 'even_gen' statement causes the generator to yield all values from the ‘even_gen’ generator (which produces even numbers) until it is exhausted. However, the ‘even_gen’ generator is an infinite generator, meaning it never exhausts and keeps producing even numbers indefinitely.
• What Happens:
• When yield from 'even_gen' is called, the 'even_odd_chain' generator yields values from ‘even_gen’ (the even numbers) continuously.
• Since 'even_gen' never stops (it's an infinite loop that keeps yielding even numbers), the yield from ‘odd_gen’ part is never reached.
• As a result, the loop in 'even_odd_chain' never moves on to the odd numbers because it is stuck continuously yielding from the infinite 'even_gen'.