In this story, I would like to raise a discussion about unit testing in data engineering. Although there are plenty of articles on Python unit testing on the internet, the topic looks a bit vague and uncovered. We will speak about data pipelines, the parts they consist of and how we can test them to ensure continuous delivery. Each step of the data pipeline can be considered as a function or process and ideally, it should be tested not only as a unit but all together, integrated into one single data flow process. I’ll try to summarize the techniques that I use often to mock, patch and test data pipelines including integration and automated tests.
Testing is a crucial part of any software development lifecycle and helps developers make sure the code is reliable and can be easily maintained in the future. Consider our data pipeline as a set of processing steps or functions. In this case, unit testing can be considered as a technique of writing tests to ensure that each unit of our code, or each step of our data pipeline doesn’t produce unintended results and is fit for purpose. Plastic Oven Tray
In a nutshell, each step…
Soft Pack Lithium Battery Data Engineer, Data Strategy and Decision Advisor, Keynote Speaker | linktr.ee/mshakhomirov | @MShakhomirov