{"slug":"air-quality-statistical-analysis-r-python","title":"Statistical Analysis of Air Quality Data in R and Python: Tools, Packages and Working Code","excerpt":"The practical toolkit for air quality data analysis — R with openair, Python with pandas and scikit-learn, working code for validation, aggregation, AQI computation, trend testing, polar plots and reproducible reporting.","date":"2026-07-26","author":"DSTechnoverse","image":"/blog-images/air-quality/hero-r-python.svg","category":"Air Quality Analysis","tags":["air quality data analysis R","openair package","Python air quality","pandas","statistical analysis","reproducible research"],"content":"Spreadsheets are where air quality analysis goes to become unreproducible. A year of hourly data for eight pollutants is over 70,000 cells; a rolling 8-hour maximum computed with a dragged formula is a defect waiting to be discovered; and when next year's data arrives, the whole thing has to be rebuilt by hand. Scripted analysis in R or Python solves all three problems at once: the method is written down, the result can be regenerated in seconds, and every step is reviewable.\n\nThis guide covers the practical toolkit — which packages do what, working code for the operations that come up on every project, and the reproducibility practices that make an analysis defensible rather than merely correct.\n\n**Statistical analysis of air quality data in R and Python uses purpose-built packages — openair in R and pandas with scipy and scikit-learn in Python — to validate monitoring data, aggregate it to regulatory averaging periods, compute compliance and AQI statistics, run non-parametric trend tests, produce diagnostic plots such as polar plots and calendar heatmaps, and generate reproducible reports directly from the raw data.**\n\n![Diurnal profile — a standard openair-style output](/blog-images/air-quality/diurnal-profile.svg)\n\n## R or Python? The Honest Answer\n\nBoth are excellent. The choice depends on what surrounds the analysis.\n\n| | R | Python |\n|---|---|---|\n| Domain packages | **openair** — purpose-built for air quality, unmatched | No direct equivalent; assemble from general libraries |\n| Statistics | Deeper and more mature, especially non-parametric methods | Strong and sufficient via scipy and statsmodels |\n| Data handling | dplyr and data.table, both excellent | pandas, excellent |\n| Machine learning | Good (tidymodels, ranger) | **Best in class** — scikit-learn, XGBoost, PyTorch |\n| Geospatial | sf, terra — very strong | geopandas, rasterio — very strong |\n| Reporting | R Markdown and Quarto, outstanding | Jupyter and Quarto, very good |\n| Production pipelines | Workable | **Better** — deployment, APIs, scheduling |\n| Learning curve for analysts | Gentler for statistics-first users | Gentler for programming-first users |\n\n**Practical recommendation.** For descriptive analysis, diagnostic plots and trend testing, R with openair will get you there faster than anything else — many operations that take fifty lines of Python are one function call. For machine-learning forecasts, production data pipelines, and integration with web applications, Python is the better home. Many teams use both, and there is no reason not to.\n\n## The R Toolkit\n\n### openair\n\nDavid Carslaw's [openair package](https://davidcarslaw.github.io/openair/) is the reason many air quality analysts use R at all. It implements, as single functions, analyses that would otherwise be substantial projects:\n\n| Function | What it does |\n|---|---|\n| `polarPlot()` | Concentration as a bivariate function of wind speed and direction — the best single source-diagnostic plot available |\n| `windRose()` | Wind rose, and `pollutionRose()` for concentration-weighted roses |\n| `timeVariation()` | Diurnal, weekday and monthly profiles with confidence intervals, in one call |\n| `calendarPlot()` | Daily values as a calendar heatmap, ideal for episode identification |\n| `TheilSen()` | Theil-Sen trend with Mann-Kendall significance, deseasonalised |\n| `smoothTrend()` | Non-parametric smoothed trend with uncertainty bands |\n| `timeAverage()` | Correct aggregation with a data-capture threshold enforced |\n| `selectByDate()` | Convenient temporal subsetting |\n| `percentileRose()` | Concentration percentiles by wind direction |\n| `trajCluster()` | HYSPLIT back-trajectory clustering |\n| `corPlot()` | Correlation matrix across pollutants |\n\nSupporting packages: **[dplyr](https://dplyr.tidyverse.org/)** and **lubridate** for data manipulation and dates, **trend** and **Kendall** for additional trend tests, **zoo** for rolling windows, **ggplot2** for custom figures, **sf** for spatial work, and **openairmaps** for interactive mapping.\n\n### A worked R example\n\nThe following covers the operations that appear on nearly every project.\n\n```r\nlibrary(openair)\nlibrary(dplyr)\nlibrary(lubridate)\n\n# ---- 1. Load and structure -------------------------------------------\naq <- read.csv(\"caaqms_station.csv\", stringsAsFactors = FALSE) %>%\n  mutate(date = ymd_hms(timestamp, tz = \"Asia/Kolkata\")) %>%\n  rename(ws = wind_speed, wd = wind_dir, pm25 = PM2.5, pm10 = PM10)\n\n# ---- 2. Validation ---------------------------------------------------\naq <- aq %>%\n  mutate(\n    pm25 = ifelse(pm25 < 0 | pm25 > 1000, NA, pm25),\n    pm10 = ifelse(pm10 < 0 | pm10 > 2000, NA, pm10),\n    # PM2.5 can never exceed PM10 - flag both as suspect\n    ratio_bad = !is.na(pm25) & !is.na(pm10) & pm25 > pm10,\n    pm25 = ifelse(ratio_bad, NA, pm25),\n    pm10 = ifelse(ratio_bad, NA, pm10),\n    # stalled anemometer: zero speed with a frozen direction\n    wd = ifelse(ws < 0.5, NA, wd)\n  )\n\n# ---- 3. Data capture -------------------------------------------------\ncapture <- aq %>%\n  mutate(month = floor_date(date, \"month\")) %>%\n  group_by(month) %>%\n  summarise(\n    n_possible = n(),\n    pm25_valid = sum(!is.na(pm25)),\n    pm25_capture = round(100 * pm25_valid / n_possible, 1)\n  )\nprint(capture)\n\n# ---- 4. Aggregation with an enforced capture threshold ---------------\ndaily <- timeAverage(aq, avg.time = \"day\", data.thresh = 75)\n\n# ---- 5. NAAQS exceedance statistics ----------------------------------\nexceed <- daily %>%\n  summarise(\n    valid_days      = sum(!is.na(pm25)),\n    annual_mean     = round(mean(pm25, na.rm = TRUE), 1),\n    exceed_days     = sum(pm25 > 60, na.rm = TRUE),\n    exceed_pct      = round(100 * exceed_days / valid_days, 1),\n    p98             = round(quantile(pm25, 0.98, na.rm = TRUE), 1)\n  )\nprint(exceed)\n\n# ---- 6. Diagnostic plots ---------------------------------------------\ntimeVariation(aq, pollutant = \"pm25\", normalise = FALSE)\npolarPlot(aq, pollutant = \"pm25\", statistic = \"mean\")\ncalendarPlot(daily, pollutant = \"pm25\", year = 2025)\npollutionRose(aq, pollutant = \"pm25\")\n\n# ---- 7. Trend test ---------------------------------------------------\nTheilSen(aq, pollutant = \"pm25\", deseason = TRUE, autocor = TRUE)\n```\n\nSeven blocks, and you have validated data, a capture statement, correct aggregation, compliance statistics, four diagnostic plots and a trend test with autocorrelation handled. That is the case for openair in a nutshell.\n\n![A station comparison box plot — the kind of output a few lines of scripted analysis produces](/blog-images/air-quality/station-boxplot.svg)\n\n## The Python Toolkit\n\n| Package | Role |\n|---|---|\n| **[pandas](https://pandas.pydata.org/)** | Data structures, time indexing, resampling, rolling windows |\n| **numpy** | Numerical operations |\n| **scipy.stats** | Statistical tests, distributions |\n| **statsmodels** | Regression, time series, seasonal decomposition |\n| **pymannkendall** | Mann-Kendall variants including seasonal and pre-whitened |\n| **[scikit-learn](https://scikit-learn.org/)** | Regression, random forests, gradient boosting, model validation |\n| **xgboost / lightgbm** | High-performance gradient boosting for forecasting |\n| **matplotlib / seaborn / plotly** | Visualisation |\n| **geopandas / rasterio** | Spatial analysis |\n| **windrose** | Wind rose plotting |\n\n### A worked Python example\n\n```python\nimport pandas as pd\nimport numpy as np\nimport pymannkendall as mk\n\n# ---- 1. Load and structure -------------------------------------------\ndf = pd.read_csv(\"caaqms_station.csv\", parse_dates=[\"timestamp\"])\ndf[\"timestamp\"] = df[\"timestamp\"].dt.tz_localize(\"Asia/Kolkata\")\ndf = df.set_index(\"timestamp\").sort_index()\n\n# ---- 2. Validation ----------------------------------------------------\ndf.loc[(df.PM25 < 0) | (df.PM25 > 1000), \"PM25\"] = np.nan\ndf.loc[(df.PM10 < 0) | (df.PM10 > 2000), \"PM10\"] = np.nan\nbad = df.PM25 > df.PM10                       # physically impossible\ndf.loc[bad, [\"PM25\", \"PM10\"]] = np.nan\ndf.loc[df.wind_speed < 0.5, \"wind_dir\"] = np.nan   # stalled anemometer\n\n# ---- 3. Data capture --------------------------------------------------\ncapture = (df.PM25.resample(\"MS\").count() /\n           df.PM25.resample(\"MS\").size() * 100).round(1)\nprint(capture)\n\n# ---- 4. Aggregation with a 75% capture rule ---------------------------\ndef daily_mean(series, thresh=0.75):\n    counts = series.resample(\"D\").count()\n    means  = series.resample(\"D\").mean()\n    return means.where(counts >= 24 * thresh)\n\npm25_daily = daily_mean(df.PM25)\n\n# ---- 5. Rolling 8-hour maximum, the correct way for O3 and CO ---------\no3_8h = df.O3.rolling(\"8h\", min_periods=6).mean()\no3_8h_max_daily = o3_8h.resample(\"D\").max()\n\n# ---- 6. Compliance statistics -----------------------------------------\nvalid = pm25_daily.dropna()\nprint({\n    \"valid_days\":  len(valid),\n    \"annual_mean\": round(valid.mean(), 1),\n    \"exceed_days\": int((valid > 60).sum()),\n    \"p98\":         round(valid.quantile(0.98), 1),\n})\n\n# ---- 7. Trend test with seasonality and autocorrelation handled -------\nmonthly = pm25_daily.resample(\"MS\").mean().dropna()\nresult = mk.seasonal_test(monthly, period=12)\nprint(f\"trend={result.trend}, p={result.p:.4f}, slope={result.slope:.2f}\")\n```\n\n### AQI computation in Python\n\nThe AQI is worth scripting once and reusing, because the breakpoint tables and averaging rules are exactly the kind of thing that gets copied wrong in spreadsheets. The methodology is set out in [how to calculate the AQI in India](/blog/how-to-calculate-aqi-india).\n\n```python\nBREAKPOINTS = {                       # CPCB National AQI\n    \"PM25\": [(0,30,0,50),(30,60,51,100),(60,90,101,200),\n             (90,120,201,300),(120,250,301,400),(250,1000,401,500)],\n    \"PM10\": [(0,50,0,50),(50,100,51,100),(100,250,101,200),\n             (250,350,201,300),(350,430,301,400),(430,2000,401,500)],\n    \"NO2\":  [(0,40,0,50),(40,80,51,100),(80,180,101,200),\n             (180,280,201,300),(280,400,301,400),(400,1000,401,500)],\n    \"SO2\":  [(0,40,0,50),(40,80,51,100),(80,380,101,200),\n             (380,800,201,300),(800,1600,301,400),(1600,3000,401,500)],\n    \"CO\":   [(0,1,0,50),(1,2,51,100),(2,10,101,200),\n             (10,17,201,300),(17,34,301,400),(34,100,401,500)],\n    \"O3\":   [(0,50,0,50),(50,100,51,100),(100,168,101,200),\n             (168,208,201,300),(208,748,301,400),(748,1500,401,500)],\n    \"NH3\":  [(0,200,0,50),(200,400,51,100),(400,800,101,200),\n             (800,1200,201,300),(1200,1800,301,400),(1800,3000,401,500)],\n}\n\ndef sub_index(pollutant, conc):\n    if conc is None or np.isnan(conc):\n        return None\n    for bp_lo, bp_hi, i_lo, i_hi in BREAKPOINTS[pollutant]:\n        if bp_lo <= conc <= bp_hi:\n            return round(i_lo + (i_hi - i_lo) / (bp_hi - bp_lo) * (conc - bp_lo))\n    return 500\n\ndef aqi(concentrations):\n    \"\"\"concentrations: dict of pollutant -> value at the correct averaging period.\"\"\"\n    subs = {p: sub_index(p, c) for p, c in concentrations.items()}\n    subs = {p: v for p, v in subs.items() if v is not None}\n    # CPCB rule: >= 3 pollutants, at least one of PM2.5 / PM10\n    if len(subs) < 3 or not ({\"PM25\", \"PM10\"} & set(subs)):\n        return None, None\n    prominent = max(subs, key=subs.get)\n    return subs[prominent], prominent\n```\n\nNote the minimum-data rule enforced in the function itself. That is the point of scripting it: the rule cannot be forgotten because it is not a step someone has to remember.\n\n## Reproducibility Practices That Matter\n\nCorrect analysis is not enough; it has to be re-runnable and reviewable.\n\n1. **Never edit the raw data.** Read it, transform it in code, write outputs elsewhere. The raw file stays pristine.\n2. **Script everything, including the plots.** A chart produced by clicking cannot be regenerated when the data is corrected.\n3. **Use version control.** Git on the analysis code, always. It is the difference between \"the numbers changed\" and \"here is the commit where the numbers changed and why.\"\n4. **Pin your environment.** `renv` in R, a lock file or `requirements.txt` in Python. Package updates silently change results.\n5. **Use literate reporting.** R Markdown, Quarto or Jupyter, so the narrative and the code producing the numbers live in the same document. Every figure in the report is generated from the data at build time, which eliminates the pasted-stale-chart failure mode entirely.\n6. **Write tests for critical functions.** The AQI function above should have a test asserting that 111 µg/m³ PM2.5 returns 268. Once written, it can never silently break.\n7. **Log the data-capture rate in the output**, not in a side note, so the reader always sees how much data supports each number.\n8. **Separate configuration from code.** Thresholds, station lists and breakpoints belong in a config file, not scattered through scripts.\n\nThese practices are covered from a general statistics perspective in our guides to [R programming for statistics](/blog/r-programming-for-statistics) and [statistical data analysis services](/blog/what-are-statistical-data-analysis-services).\n\n## Common Coding Errors in Air Quality Analysis\n\n- **Timezone mishandling.** Aggregating to \"days\" in UTC when the standard requires IST calendar days shifts every daily average by 5.5 hours.\n- **Fixed 8-hour blocks instead of a rolling maximum** for ozone and CO, which systematically under-reports.\n- **Aggregating without a capture threshold**, so a day with three valid hours becomes a daily mean.\n- **Silent NA propagation** — using `mean(x, na.rm = TRUE)` everywhere without checking how many values survived.\n- **Averaging AQI values** rather than concentrations, which is invalid because the AQI transform is non-linear.\n- **Unit confusion**, particularly CO in mg/m³ against everything else in µg/m³.\n- **Resampling before validating**, which bakes bad values into every downstream statistic.\n- **Random train-test splits on time series**, which leaks future information into the training set and produces optimistic model scores.\n\n## How DSTechnoverse Helps\n\n**DSTechnoverse** builds and runs scripted air quality analysis pipelines for consultancies, industries, municipal programmes and research groups.\n\n- **Analysis pipeline development** — reproducible R or Python pipelines covering ingestion, validation, flagging, aggregation, statistics and figure generation, delivered as code you own and can run yourself.\n- **Custom tooling** — AQI engines with the CPCB rules correctly enforced, automated NAAQS compliance reporting, exceedance trackers, and dashboard back-ends built on validated data.\n- **Analysis on demand** — where you have data and need results rather than a pipeline, we run the full analysis and deliver the report plus the scripts behind it.\n- **Code review and audit** — we review existing analysis code and spreadsheets, and routinely find timezone errors, missing capture thresholds and incorrect rolling-window logic in systems already producing published numbers.\n- **Training** — practical sessions for environmental teams on openair, pandas and reproducible reporting, using your own data.\n\nThe principle throughout is that you receive both the answer and the machinery, so next year's dataset takes an afternoon rather than a month. The same approach underpins our [statistical and research data analysis service](/services/statistical-research-data-analysis) and our [air quality analysis service](/services/air-quality-analysis).\n\n---\n\n**Need air quality data turned into a defensible answer?** DSTechnoverse provides end-to-end [air quality analysis services](/services/air-quality-analysis) — monitoring data validation, statistical analysis, source apportionment, dispersion modelling, GIS mapping, and regulator-ready reporting. [Talk to our air quality team](/contact) about your dataset or project.\n\n## Frequently Asked Questions\n\n### Should I use R or Python for air quality data analysis?\n\nUse R with openair for descriptive analysis, diagnostic plots and trend testing, where purpose-built functions do in one line what would take a page of general-purpose code. Use Python for machine-learning forecasting, production data pipelines and web integration. Many teams use both, and the file formats are interchangeable.\n\n### What is the openair package?\n\nopenair is an R package built specifically for air quality data analysis. It provides functions for polar plots, wind and pollution roses, diurnal and seasonal profiles, calendar heatmaps, Theil-Sen trend analysis with Mann-Kendall significance, correct time averaging with data-capture thresholds, and back-trajectory clustering.\n\n### How do I calculate a rolling 8-hour maximum correctly?\n\nCompute the rolling mean over every 8-hour window in the day, requiring a minimum number of valid hours in each window, then take the maximum of those rolling means. Using three fixed 8-hour blocks instead is a common error that systematically under-reports the true maximum.\n\n### Why is timezone handling important in air quality analysis?\n\nBecause regulatory averaging periods are defined on local calendar days. Aggregating in UTC when the standard requires IST shifts every daily average by five and a half hours, changing daily means, exceedance counts and the 98th-percentile value. Always localise timestamps explicitly at import.\n\n### Can air quality analysis be done in Excel?\n\nSmall compliance checks can. Anything involving a full year of hourly data across multiple parameters should not be, because rolling windows, capture thresholds and repeated aggregation are error-prone to build by hand, impossible to review, and have to be rebuilt each time new data arrives. Scripted analysis is reproducible and auditable.\n\n### What does reproducible analysis mean in practice?\n\nIt means that anyone with the raw data and the code can regenerate every number and figure in the report exactly. In practice that requires never editing raw data, scripting all transformations and plots, using version control, pinning package versions, and writing the report in a literate format where figures are generated at build time.\n\n### How do I enforce the 75% data capture rule in code?\n\nCount valid records per averaging period alongside computing the mean, then set the mean to missing wherever the count falls below the threshold. In R, openair's timeAverage has a data.thresh argument that does this directly; in Python it is a few lines combining resample().count() and resample().mean().\n\n### What is the most common mistake in air quality analysis code?\n\nAggregating before validating. Once invalid records — calibration periods, negative spikes, stuck values — are averaged into daily means, they are effectively invisible and contaminate every downstream statistic including trends, exceedance counts and AQI values. Validation must come first, always.","related":[{"slug":"air-quality-analysis-complete-guide","title":"Air Quality Analysis: The Complete Guide to Parameters, Methods and Data Interpretation"},{"slug":"air-quality-compliance-reporting-naaqs","title":"Air Quality Compliance Reporting Against NAAQS: Percentiles, Exceedances and Defensible Statistics"},{"slug":"air-quality-forecasting-machine-learning","title":"Air Quality Forecasting with Machine Learning: Models, Features and Honest Validation"},{"slug":"air-quality-gis-mapping-interpolation","title":"Air Quality GIS Mapping: Spatial Interpolation, Kriging and Exposure Surfaces"}]}