API Log Compression

Story Behind the Project

Autodesk products generate a continuous stream of API telemetry through Apigee — raw logs capturing every API call made by customers across Autodesk's product ecosystem. At ~1.4GB per hour, this data was voluminous, but most of it was noise. The real challenge wasn't storage — it was extracting the important signals and making them available at the right granularities for analytics and ML projects downstream.

The questions those downstream teams needed to answer were high-stakes: Where is API usage being abused? What does granular customer behavior actually look like? How should Autodesk think about API monetization, product subscriptions, and customer retention? None of that was possible without first transforming the raw firehose into something structured, compressed, and usable.

About

A high-performance data compression framework that processes Apigee API logs generated from customer usage of Autodesk products, compressing ~1.4GB/hr of raw telemetry by 99.8% while extracting and storing the important signals at different granularities of aggregation. The compressed, structured data feeds downstream analytics and ML pipelines for anomaly detection, customer behavior analysis, and strategic business decisions.

My Role

I was part of the team that built the foundational data engineering framework enabling this pipeline and many others at Autodesk. My contributions spanned the full lifecycle: I researched and evaluated tooling options, bringing back findings that influenced the team's decision to use Airflow DAGs and Fivetran connectors with Snowflake based on our specific use case. I wrote the SQL scripts and Python code for the Airflow DAGs that powered the framework and pipelines.

On the product side, I conducted stakeholder interviews to identify which signals mattered downstream — what data needed to be retained and at what granularities — ensuring the compression framework was built around real analytics and ML needs rather than assumptions. As the framework matured, our team templated it to support automated onboarding of additional teams leveraging Apigee for usage tracking, extending its impact across the organization.

Key Details

• 99.8% compression of Apigee API logs (~1.4GB/hr) with signal fidelity preserved across multiple aggregation granularities
• Downstream anomaly detection pipeline to identify and reduce API usage abuse
• Granular customer behavior insights enabling strategic decisions on API monetization, product subscriptions, and customer retention
• Built using Airflow DAGs, DBT, and Fivetran data connectors

Impact

• 99.8% reduction in raw API log volume (~1.4GB/hr compressed to scalable, query-ready signals)
• Enabled scalable ML pipelines for anomaly detection and granular customer behavior analysis
• Unlocked data-driven decisions on API monetization, product subscriptions, and customer retention

Technologies