Competences for the AI-Augmented Software Engineer

More and more organizations are building production-ready machine learning systems following the MLOps paradigm. In my recent publication on data engineering for machine learning systems, I argue that AI-based systems cannot exist without data. However, in that mapping study I did not find many publications on how to do proper data engineering for AI-based systems. In this post I summarize those publications that I found, tying them to the overarching framework of DataOps. We also see this shift of attention from MLOps to DataOps in the organizations we work with at Fontys ICT. To achieve real business value with AI, the data infrastructure needs to be in order first. This post defines data engineering and DataOps, and adds to that a list of “tools” that can be used for (data) engineering trustworthy AI-based systems. The post ends with a special section on data engineering for LLM-based systems.