From the free course pandas documentation on Lykke

Core Data Structures: Series and DataFrames

Key concepts: Series · DataFrame · Data Alignment · Index Labels · NaN handling

An in-depth look at the primary objects in pandas: the 1D Series and the 2D DataFrame.

Core Data Structures: Series and DataFrames

Pandas is the foundational library for data manipulation in the Python ecosystem, serving as the bridge between the raw computational power of NumPy and the structured requirements of relational databases and spreadsheets. At its heart, pandas provides two primary data structures: the Series and the DataFrame. These are not merely wrappers around arrays; they are sophisticated objects that integrate data storage, metadata (labels), and automatic alignment logic.

AI_SVGI_SVG## The Series: Labeled One-Dimensional Arrays

A Series is a one-dimensional ndarray with axis labels. While a NumPy array relies on integer-based positional indexing, a Series associates each element with a label, collectively known as the Index. Mathematically, a Series can be viewed as a mapping from a set of labels to a set of values.

Definition: A Series $S$ is a tuple $(V, I, \tau)$ where $V$ is an ordered collection of values, $I$ is an ordered set of unique (or non-unique) labels of the same length as $V$, and $\tau$ is the data type (dtype) of the elements in $V$.

Why the Series Matters

The Series solves the "context loss" problem inherent in standard arrays. In a raw array, the value 32.5 is just a float. In a Series, 32.5 can be associated with the label 2023-10-01, providing immediate temporal or categorical context. This labeling is the mechanism that enables Data Alignment, the most critical feature for handling real-world, "messy" data.

Implementation Mechanics

Internally, a Series stores its data in a contiguous block of memory (usually a NumPy ndarray or a specialized pandas ExtensionArray). The Index is maintained as a separate, highly optimized hash-map-like structure that allows for $O(1)$ lookups of values based on labels.

# Block 1: Low-level implementation and inspection
import numpy as np
import pandas as pd

# Creating a Series with explicit dtype and index
data = np.array([10, 20, 30], dtype=np.int64)
idx = ['alpha', 'beta', 'gamma']
s = pd.Series(data, index=idx, name="GreekData")

# Accessing the underlying memory and index structure
print(f"Dtype: {s.dtype}")
print(f"Values (NumPy): {s.values}")
print(f"Index object: {s.index}")

# Demonstration of label-based vs position-based access
# s.iloc[0] uses the integer offset; s.loc['alpha'] uses the hash map
assert s.iloc[0] == s.loc['alpha']

# Memory usage inspection
print(f"Memory usage (bytes): {s.memory_usage(deep=True)}")

The DataFrame: Heterogeneous Tabular Data

The DataFrame is a two-dimensional, size-mutable, and potentially heterogeneous tabular data structure. It is essentially a container for Series objects, where each Series represents a column. While the Series has one axis (the index), the DataFrame has two: the index (rows) and the columns.

Structural Properties

  1. Heterogeneity: Unlike a NumPy matrix, which requires a single dtype for the entire grid, a DataFrame allows each column to have a different dtype (e.g., one column of integers, one of strings, one of datetimes).
  2. Dual-Indexing: Operations can be performed along either axis (axis 0 for rows, axis 1 for columns).
  3. Dict-like Behavior: A DataFrame can be thought of as a dict of Series objects, where the keys are column names.
Property Series DataFrame
Dimensionality 1D 2D
Data Types Single dtype per Series Heterogeneous (per-column)
Primary Use Vectorized sequences Tabular datasets / SQL-like tables
Mutability Value mutable Value and Size mutable
Index Single Index Row Index + Column Index

The "Tidy Data" Philosophy

Pandas encourages the "Tidy Data" structure, where:

  1. Each variable forms a column.
  2. Each observation forms a row.
  3. Each type of observational unit forms a table.

AI_DEMOI_DEMO## Data Alignment: The Killer Feature

Data alignment is the process by which pandas automatically matches data from different objects based on their labels during an operation. If you add two Series, pandas does not add them by position (like NumPy); it adds them by matching labels.

The Alignment Algorithm

When an operation is performed between two objects (e.g., $S_1 + S_2$):

  1. Union of Indices: Pandas computes the union of the indices of $S_1$ and $S_2$.
  2. Reindexing: Both objects are reindexed to this new union index.
  3. NaN Injection: Labels present in one object but not the other are filled with NaN (Not a Number).
  4. Vectorized Operation: The operation is performed on the aligned values.
% Block 2: Mathematical Derivation of Alignment
Let S_1 = { (l_i, v_i) } and S_2 = { (k_j, w_j) }
The result R = S_1 \oplus S_2 is defined as:

Index(R) = I_{S1} \cup I_{S2}

Value(R)_m = 
\begin{cases} 
v_m \oplus w_m & \text{if } m \in I_{S1} \cap I_{S2} \\
\text{NaN} & \text{otherwise}
\end{cases}

Worked Example of Alignment

Imagine two sensors recording temperatures at different times. Sensor A records at 10:00 and 11:00. Sensor B records at 11:00 and 12:00.

# Block 3: Real-world usage - Alignment in Action
s_a = pd.Series([22.1, 23.5], index=['10:00', '11:00'])
s_b = pd.Series([23.8, 24.2], index=['11:00', '12:00'])

# Adding these Series aligns them by timestamp
avg_temp = (s_a + s_b) / 2

print(avg_temp)
# Output:
# 10:00     NaN
# 11:00    23.65
# 12:00     NaN
# dtype: float64

This behavior prevents "silent errors" where data from different time points or categories are accidentally combined due to shifts in the data array.

Index Labels and Selection Logic

The Index is the most misunderstood component of pandas. It is not just a row counter; it is an immutable array acting as a metadata layer. Pandas provides several ways to access data, primarily through loc (label-based) and iloc (integer-position-based).

Indexing Methods Comparison

Method Access Type Logic Use Case
.loc[] Label-based Strict matching of index/column labels. Production code, filtering by ID/Date.
.iloc[] Position-based 0-based integer indexing (like Python lists). Programmatic loops, train/test splits.
.at[] Label-based Fast access for a single scalar value. High-performance single-cell updates.
[] Hybrid Column selection or slicing. Quick interactive exploration.

Common Pitfall: The SettingWithCopyWarning

A common error for senior engineers is attempting to modify a subset of a DataFrame: df[df['A'] > 5]['B'] = 10. This often fails because df[df['A'] > 5] might return a view or a copy. Pandas cannot guarantee that the assignment will propagate back to the original DataFrame. The correct approach is always to use .loc: df.loc[df['A'] > 5, 'B'] = 10.

Handling Missing Data (NaN)

Pandas was one of the first libraries to treat missing data as a first-class citizen. It primarily uses np.nan (from the IEEE 754 floating-point specification) to represent missing values.

The Nature of NaN

  • NaN is a float. Therefore, an integer column containing a NaN will be automatically cast to float64.
  • NaN != NaN. This is a standard floating-point rule. To detect missing values, one must use pd.isna() or pd.isnull().
  • Propagation: In arithmetic, 5 + NaN = NaN. In aggregations (like .sum()), pandas defaults to ignoring NaN (treating it as 0 or simply skipping it), which differs from NumPy's default behavior.

Strategies for Missing Data

Strategy Method Description
Detection isna() / notna() Returns a boolean mask of missing values.
Removal dropna() Removes rows or columns containing missing values.
Imputation fillna() Replaces NaN with a scalar or a calculated value (mean/median).
Interpolation interpolate() Fills NaN using linear, polynomial, or time-based logic.
# Block 4: Edge cases and Nullable Types
# Traditional behavior: Integer column becomes float due to NaN
s_int = pd.Series([1, 2, None], dtype=object) # Forced to object or float
print(f"Traditional dtype with None: {pd.Series([1, 2, np.nan]).dtype}")

# Modern Pandas (1.0+): Nullable Integer Types
# Using the capital 'I' Integer dtype allows NaNs without casting to float
s_nullable = pd.Series([1, 2, None], dtype="Int64")
print(f"Nullable Integer dtype: {s_nullable.dtype}")
print(f"Is element 2 null? {s_nullable.isna()[2]}")

# Performance comparison: Vectorized fillna vs manual loop
# df.fillna(0) is orders of magnitude faster than iterating rows

Advanced Concept: MultiIndex (Hierarchical Indexing)

While DataFrames are 2D, the MultiIndex allows pandas to represent higher-dimensional data within a 2D structure. This is essentially "nesting" indices.

Why MultiIndex?

It allows for sophisticated "group-by" operations and data reshaping (pivoting). For example, a dataset tracking "Sales" across "Year" and "Region" can have a MultiIndex of ['Year', 'Region'].

Mechanics

A MultiIndex is stored as a list of levels (the unique labels for each level) and codes (integers mapping each position to a level label). This representation is extremely memory-efficient.

# Block 5: DevOps/Environment Setup for Pandas Performance
# High-performance pandas often requires specific C-extensions and engines
# Install pandas with performance extras:
pip install "pandas[performance, excel, parquet]"

# For faster I/O, ensure 'pyarrow' or 'fastparquet' is available
# For faster expression evaluation, 'numexpr' is recommended
pip install pyarrow numexpr

# Set environment variable to use the 'PyArrow' backed strings (Pandas 3.0+ default)
export PANDAS_USE_PYARROW=1

Summary of Best Practices

  1. Vectorization over Loops: Never iterate over a DataFrame with for index, row in df.iterrows() unless absolutely necessary. Use vectorized Series operations.
  2. Explicit Indexing: Use .loc and .iloc to avoid ambiguity and SettingWithCopy warnings.
  3. Dtype Management: Downcast numeric types (float64 to float32) and use category dtypes for low-cardinality strings to save up to 90% memory.
  4. Alignment Awareness: Always check indices before joining or adding Series to ensure you aren't creating unintended NaN values.

AI_STUDY_GUIDEI_STUDY_GUIDE### Further Reading

  • The "Tidy Data" paper by Hadley Wickham.
  • Pandas Internals: The BlockManager vs. the new ArrayManager.
  • IEEE 754 Floating-Point Standard and its implications for data science.
Core Data Structures: Series and DataFrames - pandas documentation - diagram 1
Core Data Structures: Series and DataFrames - pandas documentation - diagram 1

Source Materials

Study pandas documentation on Lykke

This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.

Add the free extensionSign in to open every section

All sections of pandas documentation · Browse more course pages