Why I Replaced Parts of My Python Automation Stack With Rust Extensions

When Python Hit Its Limits I’ve been building automation frameworks in Python for years — orchestrating APIs, running micro-agents, moving data, and managing workflow pipelines. Python is elegant, easy to maintain, and fast enough for most tasks. But eventually, the bottlenecks became obvious:

Heavy numeric computation in data preprocessing
Tight loops in internal ETL engines
High-frequency API polling that couldn’t keep up with sub-millisecond intervals
Serialization-heavy microservices where Python’s GIL caused subtle performance degradation

I realized I didn’t need to rewrite everything. I only needed to accelerate the critical hotspots. That’s when I decided to start embedding Rust into my Python stack.

PyO3: The Bridge Between Worlds

PyO3 is a Rust crate that makes writing Python extensions in Rust straightforward. At first, I expected FFI (foreign function interface) to be painful — but with PyO3, the process was surprisingly clean.

Basic Rust Module for Python

Here’s a simplified example of turning a small Rust function into a Python-callable module:

use pyo3::prelude::*;

/// Compute factorial recursively — pure Rust speed
#[pyfunction]
fn factorial(n: u64) -> PyResult<u64> {
    if n == 0 { Ok(1) } else { Ok(n * factorial(n - 1)?) }
}

/// Define the Python module
#[pymodule]
fn rust_ext(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(factorial, m)?)?;
    Ok(())
}
To build:
maturin develop
Then, in Python:
import rust_ext

print(rust_ext.factorial(20))
That single snippet runs orders of magnitude faster than pure Python recursion. I used this pattern for numeric-heavy and computationally intensive tasks.

Targeting Performance Hotspots

1. Data Serialization & Compression One of the pain points in my ETL workflows was frequent JSON serialization and deserialization. Python’s json module was okay for small payloads, but when processing thousands of messages per second, it became a bottleneck.

I rewrote the serialization in Rust using serde_json:
use pyo3::prelude::*;
use serde_json::Value;

#[pyfunction]
fn parse_json(input: &str) -> PyResult<Value> {
    let v: Value = serde_json::from_str(input)?;
    Ok(v)
}

#[pymodule]
fn rust_json(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(parse_json, m)?)?;
    Ok(())
}

This alone cut deserialization time in half for my largest ETL streams.

2. Fast File Processing Loops

My Python automation often touched hundreds of log files per minute. Iterating over them line by line in Python wasn’t scalable. Using Rust:

use pyo3::prelude::*;
use std::fs::File;
use std::io::{BufReader, BufRead};

#[pyfunction]
fn count_lines(path: &str) -> PyResult<usize> {
    let file = File::open(path)?;
    let reader = BufReader::new(file);
    Ok(reader.lines().count())
}

#[pymodule]
fn rust_file(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(count_lines, m)?)?;
    Ok(())
}
Python call:
import rust_file

lines = rust_file.count_lines("huge_log.log")
print(f"Lines: {lines}")

This replaced a slow Python loop and reduced IO-bound processing from minutes to seconds.

3. Async-Friendly Extensions The real magic came when combining Rust’s async ecosystem with Python’s asyncio. Using Tokio + PyO3, I can now run high-throughput async tasks without Python’s GIL limiting concurrency. Example: polling multiple endpoints concurrently in Rust while Python orchestrates:

use pyo3::prelude::*;
use pyo3_asyncio::tokio::future_into_py;
use reqwest::Client;

#[pyfunction]
fn fetch_urls(py: Python, urls: Vec<String>) -> PyResult<&PyAny> {
    let client = Client::new();

    future_into_py(py, async move {
        let mut results = Vec::new();
        for url in urls {
            let resp = client.get(&url).send().await?.text().await?;
            results.push(resp);
        }
        Ok(results)
    })
}
Python side:
import asyncio
import rust_async

urls = ["https://api.service1.com", "https://api.service2.com"]

async def main():
    responses = await rust_async.fetch_urls(urls)
    print(responses)

asyncio.run(main())

Suddenly, high-frequency API polling and microservice orchestration ran faster than I imagined — with Python still managing the async control flow.

Why This Hybrid Approach Works

Python remains the glue: orchestrating APIs, scheduling tasks, handling exceptions, logging, and managing workflows.
Rust accelerates the hotspots: CPU-intensive computation, file IO, JSON serialization, and high-throughput network polling.
Async interoperability: Python retains the event loop, while Rust handles low-level async operations efficiently.
Maintenance-friendly: only replace what matters, leaving 80–90% of code in Python.

It’s the best of both worlds.

Lessons Learned From Building FFI Bridges

Measure before replacing — not everything needs Rust. Focus only on bottlenecks.
Keep Python async-aware — make Rust extensions compatible with asyncio where concurrency matters
Use Maturin — packaging Rust extensions for Python is trivial if you adopt a consistent build process.
Logging and error handling matter — map Rust errors to Python exceptions properly to avoid silent crashes.

After implementing these lessons, I now have a hybrid stack that runs heavy workloads 3–10x faster without complicating the Python codebase.

The Takeaway: Python + Rust = Superpowers I didn’t leave Python; I augmented it. Python still orchestrates, schedules, logs, and integrates. Rust accelerates the heavy tasks. Together, they form a Python micro-automation engine that’s faster, safer, and easier to maintain than any pure-Python approach I’ve tried. For my automation workflows — ETL, API orchestration, async polling, file processing, and message-driven tasks — this hybrid approach isn’t just convenient. It’s transformative.

Read the full article here: https://medium.com/rustaceans/why-i-replaced-parts-of-my-python-automation-stack-with-rust-extensions-60b9161186f6