Python pandas Tutorial: Load, Filter, Group, and Export DataFrames

Overview

pandas is the standard library for tabular data in Python. It powers data analysis, ETL pipelines, and reporting. This tutorial covers the operations you will use most often: loading data, selecting and filtering, aggregating, and exporting results.

Install pandas

pip install pandas openpyxl

openpyxl is required to read and write Excel files. See the pandas installation guide for other options.

Loading Data

Function	Source
`pd.read_csv("file.csv")`	CSV file
`pd.read_excel("file.xlsx")`	Excel file
`pd.read_json("file.json")`	JSON file
`pd.read_sql(query, conn)`	SQL database
`pd.read_parquet("file.parquet")`	Parquet file

import pandas as pd

df = pd.read_csv("sales.csv")
df.head()
df.info()
df.describe()

Selecting Columns and Rows

# Select a column (Series)
df["amount"]

# Select multiple columns (DataFrame)
df[["region", "amount"]]

# Select by index position
df.iloc[0:5, 0:2]

# Select by label
df.loc[df["region"] == "EU", ["region", "amount"]]

Filtering

# Single condition
high_value = df[df["amount"] > 1000]

# Multiple conditions (use & and |, wrap each in parentheses)
filtered = df[(df["amount"] > 1000) & (df["region"] == "EU")]

# Membership test
df[df["category"].isin(["Electronics", "Books"])]

# Not null
df[df["discount"].notna()]

Creating and Modifying Columns

# Arithmetic
df["total"] = df["amount"] * (1 - df["discount"].fillna(0))

# Conditional column
df["tier"] = df["amount"].apply(lambda x: "high" if x > 1000 else "low")

# Vectorized with numpy
import numpy as np
df["above_avg"] = np.where(df["amount"] > df["amount"].mean(), 1, 0)

Grouping and Aggregation

summary = df.groupby("region").agg(
    total_sales=("amount", "sum"),
    avg_sale=("amount", "mean"),
    orders=("amount", "count")
).reset_index()

summary.sort_values("total_sales", ascending=False)

Aggregation	Description
`sum`	Sum of values
`mean` / `median`	Average / middle value
`count`	Number of non-null values
`nunique`	Number of unique values
`min` / `max`	Minimum / maximum

Joining DataFrames

orders = pd.read_csv("orders.csv")
customers = pd.read_csv("customers.csv")

merged = orders.merge(
    customers,
    left_on="customer_id",
    right_on="id",
    how="left"
)

The how parameter accepts inner, left, right, and outer, matching SQL join semantics.

Handling Missing Data

# Detect
df.isna().sum()

# Drop rows with any missing value
df.dropna()

# Fill with a constant
df["discount"] = df["discount"].fillna(0)

# Forward-fill time series
df["price"] = df["price"].ffill()

Pivot and Reshape

pivot = df.pivot_table(
    values="amount",
    index="region",
    columns="category",
    aggfunc="sum",
    fill_value=0
)

Exporting Data

df.to_csv("output.csv", index=False)
df.to_excel("output.xlsx", index=False, sheet_name="Report")
df.to_json("output.json", orient="records")
df.to_parquet("output.parquet")

Performance Tips

Prefer vectorized operations over apply; they run in C.
Use category dtype for low-cardinality string columns to save memory.
Read only needed columns with usecols.
For datasets larger than memory, use chunked reading or switch to Polars or DuckDB.