Compare column summation using for loop vs. sum method in Pandas

Last update on December 21 2024 07:40:50 (UTC/GMT +8 hours)

Pandas: Performance Optimization Exercise-1 with Solution

Write a Pandas program to create a large DataFrame and measure the time taken to sum a column using a for loop vs. using the sum method.

Sample Solution :

Python Code :

import pandas as pd  # Import the Pandas library
import numpy as np  # Import the NumPy library
import time  # Import the time module to measure execution time

# Create a large DataFrame with random integers
np.random.seed(0)  # Set seed for reproducibility
data = np.random.randint(1, 100, size=(1000000, 1))  # Generate random data
df = pd.DataFrame(data, columns=['Values'])  # Create a DataFrame

# Measure the time taken to sum the column using a for loop
start_time = time.time()  # Record the start time
sum_for_loop = 0  # Initialize the sum variable
for value in df['Values']:  # Iterate through each value in the column
    sum_for_loop += value  # Add the value to the sum variable
time_for_loop = time.time() - start_time  # Calculate the time taken

# Measure the time taken to sum the column using the sum method
start_time = time.time()  # Record the start time
sum_method = df['Values'].sum()  # Use the sum method to calculate the sum
time_sum_method = time.time() - start_time  # Calculate the time taken

# Print the results
print("Sum using for loop:", sum_for_loop)
print("Time taken using for loop:", time_for_loop, "seconds")
print("Sum using sum method:", sum_method)
print("Time taken using sum method:", time_sum_method, "seconds")

Output:

Sum using for loop: 49988718
Time taken using for loop: 0.1499619483947754 seconds
Sum using sum method: 49988718
Time taken using sum method: 0.0010004043579101562 seconds

Explanation:

Import Libraries:

Import the Pandas library for data manipulation.
Import the NumPy library for generating random data.
Import the time module to measure execution time.

Create a Large DataFrame:

Set a seed for reproducibility using np.random.seed(0).
Generate random integers with np.random.randint and create a large DataFrame with 1,000,000 rows and one column named 'Values'.

Measure Time Using a For Loop:

Record the start time using time.time().
Initialize a variable sum_for_loop to store the sum.
Iterate through each value in the 'Values' column using a for loop and add it to sum_for_loop.
Calculate the time taken by subtracting the start time from the current time.

Measure Time Using Sum Method:

Record the start time using time.time().
Use the Pandas sum method to calculate the sum of the 'Values' column.
Calculate the time taken by subtracting the start time from the current time.

Finally display the sum and the time taken for both the for loop and the sum method.

Python-Pandas Code Editor:

Have another way to solve this solution? Contribute your code (and comments) through Disqus.

Previous: Pandas Performence Optimization Exercises Home.
Next: Compare performance of apply vs. Vectorized operations in Pandas.