Predict the future with machine learning

Challenge

Predicting future traffic flow is very important when operating roads: the more accurate we can be and the further we can predict into the future, the better we can actively influence traffic via automatic speed regulation signs, and hence avoid costly traffic jams. Prediction provides us with critical information: There are well-established statistical modeling approaches that use current flow data that works only to a certain accuracy. But combining these approaches with more modern machine learning (ML) approaches results in better prediction.

Solution

Exploratory data analysis is performed with Jupyter, Spark, Python, and R. To allow for faster ML model training, the ML platform is equipped with a GPU that can be targeted specifically for ML training workloads.

Data pipelines are developed with Spark so that they scale well with the large amounts of traffic data being produced. Some points to highlight from the ML development are:

• Data cleaning proves to be a crucial point, since the sensor data collected is unreliable in terms of accuracy and completeness
• The features developed follow recommendations from current scientific papers that tackle similar problems
• Models are trained both with SparkML and TensorFlow. Different model types are used and compared; these use regression and classification as well as outlier detection (treating a “traffic jam” as an outlier).
• Model results are compared based on agreed-upon metrics that are chosen to be robust for highly imbalanced data, since traffic jams are very rare compared to normal traffic conditions