Saurav Jain

North Bay Python

Designing Python APIs for Data You Don’t Control - Saurav Jain - 2026

System Reliability and Error Handling: Comparing paradigms from C to Rust Result to build resilient cloud infrastructure.

Ethical Technology and Social Justice: Analyzing the intersection of AI, eugenics, and data labor to advocate for empathetic software design.

Practical Development Skills: Mastering web scraping, designing Python APIs for volatile data, and building civic tech on a budget.

Open Source and Community Building: Learning best practices for project scope management, contributing to large open-source projects, and technical communication.

Explore advanced topics in Python development, system reliability, ethical AI, and community-driven open source projects.

A deep dive into error handling paradigms (C, Exceptions, Rust Result) and the systemic failures that define modern cloud infrastructure. Learn how to build truly robust systems by understanding complex failure modes.

State of Exception(s)

A critical examination of the tech industry's ideological roots, connecting eugenics, data labor, and techno-optimism. Learn to build technology with social justice and empathy.

An Economy of Empathy

A practical framework for applying technology to local activism and civic engagement, focusing on maximizing impact with minimal resources.

Running Resistance Tech on a Shoestring

Learn to build robust Python scrapers and APIs for volatile open web data, focusing on handling silent failures and maintaining data contracts.

Designing Python APIs for Data You Don’t Control

Explore essential technical communication and crisis communication skills needed to navigate rapid technological change and social uncertainty.

Crisis (Technical) Communication: Teaching Survival Skills You Didn’t Know...

A personal journey detailing the growth of an open-source project, emphasizing community building, vulnerability, and technical contribution.

Bumbling into BeeWare: From typo-fix to core developer

A retrospective on web development, covering embedded systems, SSGs, and AI tools, showing the evolution of modern developer workflows.

While I've changed gears... I've managed to find my web development tribe

Applying feminist theory to network engineering, arguing that systemic problems (like latency) require intersectional, non-linear thinking.

What Feminist Theory & Praxis Says About Internet Networking

A case study on maintaining project scope in large open-source projects, crucial for sustainable software architecture.

No Project Scope Survives Contact with Users

North Bay Python 2026

Shift your mindset from 'Does it work now?' to 'What happens when it breaks?' to prevent silent data failures.

Define a strict data contract (using Pydantic/data classes) before scraping to ensure predictable output schema.

Implement robust data cleaning and validation (e.g., regex, explicit type conversion) to handle inconsistent web formats.

Differentiate between temporary errors (e.g., rate limiting) and structural breaking changes to guide recovery logic.

Monitor data quality metrics (e.g., completeness percentage) as a critical safeguard against silent data degradation.

Learn how to build robust Python scrapers and APIs for volatile open web data, focusing on handling silent failures and data contracts.

The open web is a massive source of valuable data, but it lacks the stability and guarantees of a formal API. Unlike controlled APIs, web data has no documentation, no versioning, and changes are silent. The biggest risk is not a loud crash, but a silent failure where the scraper continues running but returns incorrect or null data, leading to flawed downstream decisions.

Instead of simply scraping and dumping data into a dictionary, the talk advocates defining a clear data model first, using tools like Python `dataclasses` or Pydantic. This model acts as a contract, specifying which fields are required (e.g., title, URL) and which are optional (e.g., rating, review count). This ensures the output data structure is predictable, regardless of the source website's variability.

Web data is a 'disaster zone' for formatting. Pricing, for example, can appear in multiple formats (e.g., '$1,200.00' vs '€45.00'). Robust scraping requires careful handling: check for element existence before accessing it, use regular expressions for text cleaning, and use explicit type conversions (like `Decimal` instead of `float`) to maintain precision. If conversion fails, the system should log the raw text and return `None`, rather than crashing.

The system must differentiate between error types. Temporary errors (like rate limiting or CAPTCHAs) should trigger retries or proxy swaps. Structural breaking changes (e.g., a complete redesign of the product page) require human intervention. Furthermore, Pydantic is highly recommended for automatic validation, enforcing rules (e.g., rating must be 1-5) and providing clear error messages when data types or constraints are violated. The `extra='ignore'` setting helps prevent crashes when websites introduce new, unknown fields.

When providing data derived from external sources, treat your output as a formal contract. Changes to the schema (removing, renaming, or changing types) must be communicated to users well in advance (e.g., 14 business days). Finally, continuous data quality monitoring is crucial. Tracking metrics like the percentage of non-null fields is the fastest way to detect a silent failure before it impacts business intelligence.

Designing Python APIs for Data You Don’t Control - 2026

transcriptions/transcription-8jFb-oAFqTg-v1.txt

cotalks.dev

Saurav Jain

Videos