Reflections on Big Data, User Need, and vision needed to find solutions

I read a really interesting article this morning, sent to me by a friend, it was on the Conversation platform it was titled Big data has not revolutionised medicine — we need big theory alongside it and it struck me in a number of ways not least that I felt it was spot on as an article observing how confused people get about the use of Big Data. The sense of fashion over style and the search for the silver bullet that removes the need for human involvement.

There are parallels such as the difference, in my view, between the concept of “start with user need” and having the vision to see a future based on a hypothesis you need to prove.

I am sure no one wants to to die of cancer do they? so this by definition is equals “user need”. How to prevent or cure cancer that must equal “vision, hypothesis, research, experimentation, innovation” all unlikely to come out of the mouth of the user expressed as a solution is it.

Those that get agile development know this, unfortunately there are many who know very little but continue to spout on about such things without understanding the full picture or approach.

Sure data big and small can inform and be used to test hypothesis, sure software can perform analysis fast and discover unseen patterns in the haystack but it takes the germ of an idea to target that investigation, the article makes a good point about the risks and limitations of training data and the need for starting assumptions not underpinned by data, ideas, vision, innovation, inspiration.

Sure we must be agile, we must adapt, we should seek to fail fast, to try new things but where you start and why you start where you do, that is the stuff of dreams, instinct, experience, passion and the wonder of human synaptic pathways firing in ways we still don’t understand and cannot easily replicate

As much as I value silicon and binary based technology and artifical intelligence, machine learning in all its forms, we still need carbon based life forms, organic ones prepared to dream and have the Eureka moment