Deep learning in space and time
A05 aims for a better theoretical understanding of deep neural networks for space-time data, which will result in improved predictions for spatio-temporal data including statistical guarantees. In the long run, deep-learning approaches will be developed to meet demands such as interpretability, statistical inference and statistical optimality.aims for a better theoretical understanding of deep neural networks for space-time data, which will result in improved predictions for spatio-temporal data including statistical guarantees. In the long run, deep-learning approaches will be developed to meet demands such as interpretability, statistical inference and statistical optimality.
Project Leaders
Prof. Dr. Asja Fischer
Faculty of Computer Science - Chair of Machine Learning
Ruhr University Bochum
Prof. Dr. Johannes Lederer
Department of Mathematics - Chair of Mathematics of Data-Driven Methods
University of Hamburg
Prof. Dr. Hanna Meyer
Faculty of Geosciences - Chair of Remote Sensing and Spatial Modelling
University of Münster
Summary
Deep learning has great potential for spatio-temporal data due to its ability to deal with complex patterns in such data sets, enabling a comprehensive modeling of dynamic processes. However, the integration of information in both spatial and temporal dimensions into a unified and comprehensive model is still challenging and requires further insight and exploration, both theoretically as well as for practical application. Our goal is a better statistical understanding of these models and the development of an efficient framework for reliable spatio-temporal predictions. We will develop and test deep-learning frameworks concentrating on recurrent space-time graph neural networks, which model the features as nodes of a graph that can exchange information over time. We analyze the individual components of this framework and how these components act in concert to model data in space and time. Our sharp focus on this modeling approach allows us to work on the entire statistical pipeline from theory to applications. Our theoretical contributions are prediction guarantees, accounting in particular for sparsity-inducing regularization, and improvements regarding the transferability of the trained models. We believe that theory and application should be integrated to mutually benefit each other. Therefore, on the application side, we study earth-observation satellite images as a prototype for dense spatio-temporal data, which will serve as a testbed for analyzing the developed methods. This allows us to ensure theoretical as well as practical relevance of our project, to draw on the expertise of the three project leaders, and to connect to many theoretical and applied projects of TRR 391.
Adilova, L., B. C. Geiger, and A. Fischer (2023). Information Plane Analysis for Dropout Neural Networks. The Eleventh International Conference on Learning Representations. url: https://openreview.net/forum?id=bQB6qozaBw.
Anthony, M. and P. L. Bartlett (2009). Neural network learning: Theoretical foundations. Cambridge University Press.
Baroud, S., S. Chokri, S. Belhaous, and M. Mestari (2021). A Brief Review of Graph Convolutional Neural Network Based Learning for classifying remote sensing images. Procedia Computer Science 191, 349–354. doi: 10.1016/j.procs.2021.07.047.
Binder, N., H. Dette, J. Franz, D. Zöller, et al. (2022a). Data Mining in Urology: Understanding Real-world Treatment Pathways for Lower Urinary Tract Systems via Exploration of Big Data. European urology focus. doi: 10.1016/j.euf.2022.03.019.
Binder, N., T. A. Gerds, and P. K. Andersen (2014). Pseudo-observations for competing risks with covariate dependent censoring. Lifetime data analysis 20, 303–315. doi: 10.1007/s10985-013-9247-7.
Binder, N., A.-S. Herrnböck, and M. Schumacher (2017). Estimating hazard ratios in cohort data with missing disease information due to death. Biometrical Journal 59, 251–269. doi: 10.1002/bimj.201500167.
Binder, N., K. Möllenhoff, A. Sigle, and H. Dette (2022b). Similarity of competing risks models with constant intensities in an application to clinical healthcare pathways involving prostate cancer surgery. Statistics in Medicine 41, 3804–3819. doi: 10.1002/sim.9481.
Bojchevski, A. and S. Günnemann (2019). Certifiable Robustness to Graph Perturbations. arXiv. url: http://arxiv.org/abs/1910.14356.
Bretz, F., K. Möllenhoff, H. Dette, W. Liu, et al. (2018). Assessing the similarity of dose response and target doses in two non-overlapping subgroups. Statistics in Medicine 37, 722–738. doi: 10.1002/sim.7546.
Bunea, F., J. Lederer, and Y. She (2013). The Group Square-Root Lasso: Theoretical Properties and Fast Algorithms. doi: 10.1109/tit.2013.2290040.
Cao, D., Y. Wang, J. Duan, C. Zhang, et al. (2020). Spectral Temporal Graph Neural Network for Multivariate Time-Series Forecasting. Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS’20. Red Hook, NY, USA: Curran Associates Inc. url: https://dl.acm.org/doi/10.5555/3495724.3497215.
Chen, H., T. Furuya, S. Fukagai, S. Saga, et al. (2020). Wheel slip/Slide and low adhesion caused by fallen leaves. Wear 446–447, 203187. doi: 10.1016/j.wear.2020.203187.
Chu, P., J. Wang, Q. You, H. Ling, et al. (2023). TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 4859–4869. doi: 10.1109/WACV56688.2023.00485.
Cohen, T. and M. Welling (2016). Group Equivariant Convolutional Networks. Proceedings of The 33rd International Conference on Machine Learning. Ed. by M. F. Balcan and K. Q. Weinberger. Vol. 48. Proceedings of Machine Learning Research. PMLR, 2990–2999. url: https://proceedings.mlr.press/v48/cohenc16.html.
Dalalyan, A. S., M. Hebiri, and J. Lederer (2017). On the prediction performance of the Lasso. Bernoulli 23. doi: 10.3150/15-BEJ756.
Damm, S., D. Forster, D. Velychko, Z. Dai, et al. (2023). The ELBO of Variational Autoencoders Converges to a Sum of Entropies. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics. Ed. by F. Ruiz, J. Dy, and J.-W. van de Meent. Vol. 206. Proceedings of Machine Learning Research. PMLR, 3931–3960. url: https://proceedings.mlr.press/v206/damm23a.html.
Däubener, S. and A. Fischer (2022). How Sampling Impacts the Robustness of Stochastic Neural Networks. Advances in Neural Information Processing Systems. Ed. by A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho. url: https://openreview.net/forum?id=-zBN5sBzdvr.
Dette, H., K. Möllenhoff, S. Volgushev, and F. Bretz (2018). Equivalence of regression curves. Journal of the American Statistical Association 113, 711–729. doi: 10.1080/01621459.2017.1281813.
Gold, D., J. Lederer, and J. Tao (2020). Inference for high-dimensional instrumental variables regression. Journal of Econometrics 217, 79–111. doi: 10.1016/j.jeconom.2019.09.009.
Goodfellow, I., Y. Bengio, and A. Courville (2016). Deep learning. MIT press.
Guo, S., Y. Lin, N. Feng, C. Song, et al. (2019). Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33, 922–929. doi: 10.1609/aaai.v33i01.3301922.
Hastie, T., R. Tibshirani, and M. Wainwright (2015). Statistical learning with sparsity: The lasso and generalizations. CRC press.
He, S., F. Bastani, S. Jagwani, E. Park, et al. (2020). RoadTagger: Robust Road Attribute Inference with Graph Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence 34, 10965–10972. doi: 10.1609/aaai.v34i07.6730.
Hebiri, M. and J. Lederer (2020). Layer Sparsity in Neural Networks. url: http://arxiv.org/abs/2006.15604.
Huang, S.-T. and J. Lederer (2023). DeepMoM: Robust Deep Learning With Median-of-Means. Journal of Computational and Graphical Statistics 32, 181–195. doi: 10.1080/10618600.2022.2090947.
Jegelka, S. (2022). Theory of Graph Neural Networks: Representation and Learning.
Karimi, A. M., Y. Wu, M. Koyuturk, and R. H. French (2021). Spatiotemporal Graph Neural Network for Performance Prediction of Photovoltaic Power Systems. Proceedings of the AAAI Conference on Artificial Intelligence 35, 15323–15330. doi: 10.1609/aaai.v35i17.17799.
Kattenborn, T., J. Eichel, and F. E. Fassnacht (2019). Convolutional Neural Networks enable efficient, accurate and fine-grained segmentation of plant species and communities from high-resolution UAV imagery. Scientific Reports 9, 17656. doi: 10.1038/s41598-019-53797-9.
Kattenborn, T., F. Schiefer, J. Frey, H. Feilhauer, et al. (2022). Spatially autocorrelated training and validation samples inflate performance assessment of convolutional neural networks. ISPRS Open Journal of Photogrammetry and Remote Sensing 5, 100018. doi: 10.1016/j.ophoto.2022.100018.
Kim, B.-H., J. C. Ye, and J.-J. Kim (2021). Learning Dynamic Graph Representation of Brain Connectome with Spatio-Temporal Attention. Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan. doi: 10.48550/arXiv.2105.13495.
Kipf, T. N. and M. Welling (2016). Semi-Supervised Classification with Graph Convolutional Networks.
Kohl, T., A. Sigle, T. Kuru, J. Salem, et al. (2022). Comprehensive analysis of complications after transperineal prostate biopsy without antibiotic prophylaxis: Results of a multicenter trial with 30 days’ follow-up. Prostate Cancer and Prostatic Diseases 25, 264–268. doi: 10.1038/s41391-021-00423-3.
Krause, O., A. Fischer, T. Glasmachers, and C. Igel (n.d.). Approximation properties of DBNs with binary hidden units and real-valued visible units.
Lary, D. J., A. H. Alavi, A. H. Gandomi, and A. L. Walker (2016). Machine learning in geosciences and remote sensing. Geoscience Frontiers 7, 3–10. doi: 10.1016/j.gsf.2015.07.003.
Lauritzen, S. L. (1996). Graphical models. Vol. 17. Clarendon Press.
Lederer, J. (2021). Activation Functions in Artificial Neural Networks: A Systematic Overview.
Lederer, J. (2022). Fundamentals of high-dimensional statistics: With Exercises and R labs. Springer. doi: 10.1007/978-3-030-73792-4.
Lederer, J. (2023a). A First Course in Statistical Learning: With Data Examples and Python Code. Statistics and Computing. Springer.
Lederer, J. (2023b). Statistical guarantees for sparse deep learning. AStA Advances in Statistical Analysis, 1–28. doi: 10.1007/s10182-022-00467-3.
Li, Q., Z. Han, and X.-m. Wu (2018). Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. doi: 10.1609/aaai.v32i1.11604.
Li, Y., D. Tarlow, M. Brockschmidt, and R. Zemel (2017). Gated Graph Sequence Neural Networks. url: http://arxiv.org/abs/1511.05493.
Liang, S., A. Zhao, M. Qin, L. Hu, et al. (2023). A Graph Memory Neural Network for Sea Surface Temperature Prediction. Remote Sensing 15. doi: 10.3390/rs15143539.
Linnenbrink, J., C. Milà, M. Ludwig, and H. Meyer (2023). kNNDM: k-fold Nearest Neighbour Distance Matching Cross-Validation for map accuracy estimation. EGUsphere 2023, 1–16. doi: 10.5194/egusphere-2023-1308.
Ludvigsen, J. and R. Klæboe (2014). Extreme weather impacts on freight railways in Europe. Natural Hazards 70, 767–787. doi: 10.1007/s11069-013-0851-3.
Ludwig, M., A. Moreno-Martinez, N. Hölzel, E. Pebesma, et al. (2023). Assessing and improving the transferability of current global spatial prediction models. Global Ecology and Biogeography 32, 356–368. doi: 10.1111/geb.13635.
Ludwig, M., T. Morgenthal, F. Detsch, T. P. Higginbottom, et al. (2019). Machine learning and multi-sensor based modelling of woody vegetation in the Molopo Area, South Africa. Remote Sensing of Environment 222, 195–203. doi: 10.1016/j.rse.2018.12.019.
Ma, L., Y. Liu, X. Zhang, Y. Ye, et al. (2019). Deep learning in remote sensing applications: A meta-analysis and review. ISPRS Journal of Photogrammetry and Remote Sensing 152, 166–177. doi: 10.1016/j.isprsjprs.2019.04.015.
Maron, H., H. Ben-Hamu, N. Shamir, and Y. Lipman (2019). Invariant and Equivariant Graph Networks. url: http://arxiv.org/abs/1812.09902.
Meyer, H., M. Kühnlein, T. Appelhans, and T. Nauss (2016). Comparison of four machine learning algorithms for their applicability in satellite-based optical rainfall retrievals. Atmospheric Research 169, 424–433. doi: 10.1016/j.atmosres.2015.09.021.
Meyer, H., C. Milà, M. Ludwig, and J. Linnenbrink (2023). CAST: ’caret’ Applications for Spatial-Temporal Models. url: https://CRAN.R-project.org/package=CAST.
Meyer, H. and E. Pebesma (2021). Predicting into unknown space? Estimating the area of applicability of spatial prediction models. Methods in Ecology and Evolution 12, 1620–1633. doi: 10.1111/2041-210X.13650.
Meyer, H. and E. Pebesma (2022). Machine learning-based global maps of ecological variables and the challenge of assessing them. Nature Communications 13, 2208. doi: 10.1038/s41467-022-29838-9.
Meyer, H., C. Reudenbach, T. Hengl, M. Katurji, et al. (2018). Improving performance of spatio-temporal machine learning models using forward feature selection and target-oriented validation. Environmental Modelling & Software 101, 1–9. doi: 10.1016/j.envsoft.2017.12.001.
Meyer, H., C. Reudenbach, S. Wöllauer, and T. Nauss (2019a). Importance of spatial predictor variable selection in machine learning applications – Moving from data reproduction to spatial prediction. Ecological Modelling 411, 108815. doi: 10.1016/j.ecolmodel.2019.108815.
Meyer, H., J. Schmidt, F. Detsch, and T. Nauss (2019b). Hourly gridded air temperatures of South Africa derived from MSG SEVIRI. International Journal of Applied Earth Observation and Geoinformation 78, 261–267. doi: 10.1016/j.jag.2019.02.006.
Micheli, A. (2009). Neural Network for Graphs: A Contextual Constructive Approach. IEEE Transactions on Neural Networks 20, 498–511. doi: 10.1109/tnn.2008.2010350.
Milà, C., J. Mateu, E. Pebesma, and H. Meyer (2022). Nearest neighbour distance matching Leave-One-Out Cross-Validation for map validation. Methods in Ecology and Evolution 13, 1304–1316. doi: 10.1111/2041-210X.13851.
Möllenhoff, K., F. Bretz, and H. Dette (2020). Equivalence of regression curves sharing common parameters. Biometrics 76, 518–529. doi: 10.1111/biom.13149.
Möllenhoff, K., H. Dette, E. Kotzagiorgis, S. Volgushev, et al. (2018). Regulatory assessment of drug dissolution profiles comparability via maximum deviation. Statistics in medicine 37, 2968–2981. doi: 10.1002/sim.7689.
Möllenhoff, K., F. Loingeville, J. Bertrand, T. T. Nguyen, et al. (2022). Efficient model-based bioequivalence testing. Biostatistics 23, 314–327. doi: 10.1093/biostatistics/kxaa026.
Nicolicioiu, A., I. Duta, and M. Leordeanu (2019). Recurrent Space-time Graph Neural Networks. arXiv. url: http://arxiv.org/abs/1904.05582.
Petzka, H., A. Fischer, and D. Lukovnikov (2017). On the regularization of Wasserstein GANs. Vol. 6.
Qiu, J., J. Tang, H. Ma, Y. Dong, et al. (2018). DeepInf: Social Influence Prediction with Deep Learning. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2110–2119. doi: 10.1145/3219819.3220077.
Roberts, D. R., V. Bahn, S. Ciuti, M. S. Boyce, et al. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography 40, 913–929. doi: 10.1111/ecog.02881.
Rong, Y., W. Huang, T. Xu, and J. Huang (2020). DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. url: http://arxiv.org/abs/1907.10903.
Scarselli, F., M. Gori, Ah Chung Tsoi, M. Hagenbuchner, et al. (2009). Computational Capabilities of Graph Neural Networks. IEEE Transactions on Neural Networks 20, 81–102. doi: 10.1109/TNN.2008.2005141.
Scarselli, F., A. C. Tsoi, and M. Hagenbuchner (2018). The Vapnik–Chervonenkis dimension of graph and recursive neural networks. Neural Networks 108, 248–259. doi: 10.1016/j.neunet.2018.08.010.
Schmidt-Hieber, J. (2020). Nonparametric regression using deep neural networks with ReLU activation function. The Annals of Statistics 48, 1916–1921. doi: 10.1214/19-AOS1875.
Seo, Y., M. Defferrard, P. Vandergheynst, and X. Bresson (2016). Structured Sequence Modeling with Graph Convolutional Recurrent Networks.
Shi, X., Z. Chen, H. Wang, D.-Y. Yeung, et al. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Proceedings of the 28th International Conference on Neural Information Processing Systems — Volume 1. NIPS’15. Cambridge, MA, USA: MIT Press, 802–810. doi: 10.48550/arXiv.1506.04214.
Shuman, D. I., S. K. Narang, P. Frossard, A. Ortega, et al. (2013). The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE Signal Processing Magazine 30, 83–98. doi: 10.1109/msp.2012.2235192.
Simeunović, J., B. Schubnel, P.-J. Alet, and R. E. Carrillo (2022). Spatio-Temporal Graph Neural Networks for Multi-Site PV Power Forecasting. IEEE Transactions on Sustainable Energy 13, 1210–1220. doi: 10.1109/TSTE.2021.3125200.
Taheri, M., F. Xie, and J. Lederer (2021). Statistical guarantees for regularized neural networks. Neural Networks 142, 148–161. doi: 10.1016/j.neunet.2021.04.034.
Taheri, M., F. Xie, and J. Lederer (2022). Statistical Guarantees for Approximate Stationary Points of Simple Neural Networks. doi: 10.1016/j.neunet.2021.04.034.
Wadoux, A. M. J.-C., G. B. M. Heuvelink, S. d. Bruin, and D. J. Brus (2021). Spatial cross-validation is not the right way to evaluate map accuracy. Ecological Modelling 457, 109692. doi: 10.1016/j.ecolmodel.2021.109692.
Wu, S., Y. Tang, Y. Zhu, L. Wang, et al. (2019). Session-Based Recommendation with Graph Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33, 346–353. doi: 10.1609/aaai.v33i01.3301346.
Wu, Z., S. Pan, F. Chen, G. Long, et al. (2021). A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems 32, 4–24. doi: 10.1109/TNNLS.2020.2978386.
Wu, Z., S. Pan, G. Long, J. Jiang, et al. (2020). Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’20. New York, NY, USA: Association for Computing Machinery, 753–763. doi: 10.1145/3394486.3403118.
Xu, K., W. Hu, J. Leskovec, and S. Jegelka (2019). How Powerful are Graph Neural Networks? url: http://arxiv.org/abs/1810.00826.
Yan, S., Y. Xiong, and D. Lin (2018). Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition.
Yao, L., C. Mao, and Y. Luo (n.d.). Graph Convolutional Networks for Text Classification ().
Yu, B., H. Yin, and Z. Zhu (2018). Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. doi: 10.24963/ijcai.2018/505.
Zamzam, A. S. and N. D. Sidiropoulos (2020). Physics-Aware Neural Networks for Distribution System State Estimation. IEEE Transactions on Power Systems 35, 4347–4356. doi: 10.1109/TPWRS.2020.2988352.
Zhang, J., X. Shi, J. Xie, H. Ma, et al. (2018). GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs.
Zhang, M., Y. Sun, C. Feng, Z. Zhen, et al. (2022). Graph Neural Network based Short-term Solar Irradiance Forcasting Model Considering Surrounding Meteorological Factors. 2022 IEEE/IAS 58th Industrial and Commercial Power Systems Technical Conference (I and CPS), 1–9. doi: 10.1109/ICPS54075.2022.9773879.
Zhang, X., P. Cui, R. Xu, L. Zhou, et al. (2021a). Deep Stable Learning for Out-Of-Distribution Generalization.
Zhang, X., C. Huang, Y. Xu, L. Xia, et al. (2021b). Traffic Flow Forecasting with Spatial-Temporal Graph Diffusion Network. Proceedings of the AAAI Conference on Artificial Intelligence 35, 15008–15015. doi: 10.1609/aaai.v35i17.17761.
Zhao, L. and L. Akoglu (2020). PairNorm: Tackling Oversmoothing in GNNs. arXiv. url: http://arxiv.org/abs/1909.12223.
Zhou, J., G. Cui, Z. Zhang, C. Yang, et al. (2018). Graph Neural Networks: A Review of Methods and Applications.
Zhou, K., Y. Dong, W. S. Lee, B. Hooi, et al. (2021). Effective Training Strategies for Deep Graph Neural Networks.
Zhu, X. X., D. Tuia, L. Mou, G.-S. Xia, et al. (2017). Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources. IEEE Geoscience and Remote Sensing Magazine 5, 8–36. doi: 10.1109/MGRS.2017.2762307.
Zitnik, M., M. Agrawal, and J. Leskovec (2018). Modeling Polypharmacy Side Effects with Graph Convolutional Networks. doi: 10.1093/bioinformatics/bty294.
Zügner, D., A. Akbarnejad, and S. Günnemann (2018). Adversarial Attacks on Neural Networks for Graph Data. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. London United Kingdom: ACM, 2847–2856. doi: 10.1145/3219819.3220078.
Zügner, D. and S. Günnemann (2019). Certifiable Robustness and Robust Training for Graph Convolutional Networks. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Anchorage AK USA: ACM, 246–256. doi: 10.1145/3292500.3330905.