Data cleaning
The dataset contained gaps, inconsistent timestamps and unusual observations. I developed a documented cleaning process before performing the statistical analysis.
Back to projects
Data/2026/Complete
Weather & Air Quality Research
An independent research project analysing 14,000+ hourly observations to investigate how weather conditions relate to air quality.
StatisticsResearchPython
Built with
PythonpandasNumPyMatplotlibStatisticsPearson CorrelationLinear Regression
Overview
An independent research project investigating relationships between weather conditions and air quality using more than 14,000 hourly observations. The project combines data cleaning, Pearson correlation, linear regression and statistical visualisation in Python.
What I built
I wanted to take the statistics I was learning in school and apply them to a problem where the answer was not already sitting at the back of a textbook.
I analysed more than **14,000 hourly observations** to investigate whether meteorological conditions were associated with changes in air quality.
The project involved much more than calculating a correlation coefficient.
I had to clean the data, investigate unusual observations, visualise relationships, fit regression models, inspect residuals and critically evaluate whether the results actually supported the conclusions I wanted to make.
The final work became a full research report and poster.
The problem
Air quality changes constantly.
Temperature, humidity, wind, rainfall and other environmental variables can all change at the same time, making it difficult to determine which relationships are actually meaningful.
I wanted to investigate the relationships statistically rather than simply looking at graphs and deciding that two variables "looked related."
Why I built it
This project grew directly out of my interest in statistics.
I wanted to move beyond classroom datasets where the variables are already clean and the appropriate statistical method is obvious.
Working with real-world data forced me to deal with missing values, unusual observations, assumptions and confounding factors.
That was where the statistics became much more interesting.
Engineering
Correlation analysis
Pearson correlation was used to quantify the strength and direction of linear relationships between meteorological variables and air-quality measurements.
Linear regression
Regression models were fitted to investigate relationships more formally, with residuals inspected rather than relying only on the reported coefficient.
Visual analysis
Correlation heatmaps, regression plots, residual plots and time-series visualisations were used to identify patterns and communicate the results.
Critical evaluation
The project explicitly considered correlation versus causation, confounding variables and limitations in the dataset rather than presenting statistical relationships as proof of causality.
Mathematics
Pearson correlation coefficient
Measures the strength and direction of a linear relationship between two variables.
Results
14,000+
Hourly observations analysed
Correlation + Regression
Statistical methods
32 pages
Research report
What broke
Real data is messy
A large amount of the work happened before the statistical tests. Missing observations, timestamp issues and unusual readings all had to be dealt with carefully because cleaning decisions can change the final result.
Avoiding conclusions the data cannot support
Finding a correlation is easy. Explaining what that correlation actually means is much harder. I had to distinguish between statistical association and causation and be honest about what the dataset could not establish.
What I learned
The dataset determines the quality of the analysis
A sophisticated statistical method cannot rescue poorly understood or poorly cleaned data.
Checking assumptions matters
Running a regression model is straightforward. Understanding whether the model is appropriate is where the real analytical thinking happens.
Real-world variables rarely move independently
Time of day, season, weather and human activity can all interact, making real-world statistical analysis much more complicated than isolated textbook examples.
Next
- Use multiple regression to control for additional variables
- Investigate lagged relationships between weather and pollution
- Extend the dataset across multiple years
- Compare multiple geographic locations
- Explore non-linear models