Some years ago I worked in consulting. Our sales team had sold a huge project to a client. In the first iteration, we build an ensemble data science model predicting some very important stuff, that would completely change the way they handled their operations. The client was excited! And wanted more.
So the next project that was sold was the implementation of it all. We decided to upgrade their stack with ML capabilities, and added Databricks. Back then, Databricks couldn't do all the stuff it does now, so we worked tirelessly to hack a lot of stuff around it. We implemented all ideas in code, and it turned out quite nicely!
Now our models could run on all their data and predict the strategically important thing they wanted. There was just one small problem... Because the organization was stuck in two big failing IT migrations (workspace and blob storage), the board had decided that NOBODY was allowed to add any new integrations across the entire IT landscape.
We pitched our use cases and explained that "we just needed a few simple database connections", but it was not allowed. Period. So there we had it. A fully fledged ML platform, without any data. The project leadership decided that data should be uploaded manually as csv-files, to make it all work and deliver the value that was needed.
Following orders, we added a feature that automatically triggered the ML pipeline when a new csv-file was uploaded... And that was the end of it. The director of the department we were working for left the organization. The project never continued. We left an isolated state-of-the-art Databricks ML platform behind, floating around as a ghost ship.
All those years of work for nothing. What a waste of resources 🤷🏽♂️
I am trying something new: 𝗧𝗲𝗿𝗿𝗶𝗯𝗹𝗲 𝗧𝗵𝘂𝗿𝘀𝗱𝗮𝘆𝘀. Each Thursday, I will share a story of a data project gone terribly wrong. Not to hate, but to appreciate how complex the work can be in our discipline. And also to learn from it, support each other, and just have a laugh.
Want your story shared? Send it to me and I can publish it... anonymously!