Optimize string operations in Pandas: str accessor vs. apply

Last update on December 21 2024 07:41:01 (UTC/GMT +8 hours)

Pandas: Performance Optimization Exercise-15 with Solution

Write a Pandas program to optimize the performance of string operations on a DataFrame column by using the str accessor vs. applying a custom function with apply.

Sample Solution :

Python Code :

# Import necessary libraries
import pandas as pd
import time

# Create a sample DataFrame
num_rows = 1000000
df = pd.DataFrame({
    'text': ['example_string'] * num_rows
})

# Measure time for str accessor method
start_time = time.time()
str_accessor_result = df['text'].str.upper()
end_time = time.time()
str_accessor_time = end_time - start_time

# Define a custom function to apply
def to_upper(text):
    return text.upper()

# Measure time for apply method
start_time = time.time()
apply_result = df['text'].apply(to_upper)
end_time = time.time()
apply_time = end_time - start_time

# Print the time taken for each method
print(f"Time taken using str accessor: {str_accessor_time:.6f} seconds")
print(f"Time taken using apply method: {apply_time:.6f} seconds")

Output:

Time taken using str accessor: 0.181023 seconds
Time taken using apply method: 0.139029 seconds

Explanation:

Import Libraries:

Import pandas, numpy, and time.

Create DataFrame:

Generate a sample DataFrame with 1,000,000 rows, each containing a string.

Time Measurement for str Accessor:

Measure the time taken to convert strings to uppercase using the str.upper accessor.

Define Custom Function:

Define a custom function to convert strings to uppercase.

Time Measurement for apply Method:

Measure the time taken to apply the custom function using the apply method.

Finally print the time taken for each method.

Python-Pandas Code Editor:

Have another way to solve this solution? Contribute your code (and comments) through Disqus.

Previous: Performance comparison of cumulative Sum calculation in Pandas.
Next: Reshaping DataFrame in Pandas: pivot_table vs. manual Loop.