There is no doubt that big data is going to be one of the most important tools that will assist human society in the future. Our increasingly complex society has been able to move forward, and it will continue to do so, based on rational, scientific facts and figures within the context of the needs of humanity.

As an example, neuroscience is giving us more insight into ourselves, and we are learning that many of the elements that we have always thought of as being uniquely human are based on neurological/biological processes that can be put into algorithms. The more we know, the more interesting the question is—what makes us human? And, given the progress being made in artificial intelligence (AI), this is an important question.

Society lost touch with its people

We largely still trust that our governments are guiding us through these ever-changing developments, and it must be said that the democratic processes that have ruled us since WWII are very beneficial to mankind. Disputes generally are settled with the assistance of our democratic institutions, using common sense, scientific facts, statistical information and so on.

One could argue that the underlying data and information used in these processes are used for the common good.

However, over the last 30 years, a large proportion of the population has not seen the positive results from their political systems that they had hoped for, and they are now rebelling against politicians who are increasingly using facts and figures for their own political purposes.

Selective use of that information is creating a dangerous breakdown of trust right through our democratic system. And President Trump is going one step further—he is actively undermining several of the institutions that underpin American democracy.

But it is not just government. The Volkswagen car manufacturer scandal in Germany showed a gross misuse of data for the sole purpose of making profit at the expense of the environment. Then there are the blatant intrusions into personal data by intelligence services, social media, etc.

Another very dangerous situation is occurring in China with the introduction of the Citizen Score. This form of mass surveillance and mass manipulation is one of the worst Big Brother-like scenarios that one can imagine.

Despite this misuse of big data, it will have to be reason, facts and statistics that will guide us through the many social, environmental and economic challenges that society is facing. But it is crucial that this takes place within the structures of our democratic principles as well as within our emotional and other ‘soft’ values.

So far big data has mainly been used for commercial purposes, for sometimes questionable intelligence activities, and for downright criminal activities (hacking, stealing, political interference and so on).

There is an urgent need for big data to be used for the common good. A rapid rebalancing is needed that will see big data being used for the benefit of our society. We shouldn’t be put off by its misuse and bury our heads in the sand, hoping it will go away—or, as the new conservative forces in politics would have us believe, that the answer lies in returning to the way things were in ‘the good old days’.

Big data for the common good

We should face the big data challenges head-on. Universities in Germany and the Netherlands launched the Data for Humanity Initiative, encouraging people and organisations to use the following principles:

• Do no harm
• Use data to help create peaceful coexistence
• Use data to help vulnerable people and people in need
• Use data to preserve and improve the natural environment
• Use data to help create a fair world without discrimination

New regulations and legislation might be needed to ensure that big data is used for the common good, and that it takes privacy and human rights issues equally seriously. At present most of the big data is in the hands of corporations who have shown little interest in the common good; and most of their big data activities are clouded in secrecy and used to gain competitive advantage. Just recently I also mentioned the work of Yuval Nora Harari, who warns of big data dictatorship if we don’t get this right.

One of the first critical areas will be healthcare. New medical innovations will make it possible for people to obtain information about potential illnesses they might contract, and personalised big data solutions will be on offer to mitigate this and create better health and lifestyle outcomes.

Personal benefits in the healthcare sector could be enormous, and as a result, people may be less concerned about their personal data. But the reality is that the availability of this information could be used in a positive and a negative way. The latter could lead to discrimination by insurance companies and governments. Also, different cultures might look for different outcomes—what leeway will there be for them?

With predictive analytics and complex algorithms, allowance must be made for error, and there needs to be a system of fairness in place to guide this.

What this all means is that a key principle should be for the ownership of all personal data to rest with the individual person, and that they can decide to share that information, or not, on a permission-based footing.

We have been recommending the above approach for the last two decades (but, I must say, without much success).

I can see situations where an opt-out rather than an opt-in system could be a more effective or efficient option, but that would necessitate a restoration of trust in the political system that guides such decisions.

Rather than relying on the organisations that are currently leading the development of big data (Google, Facebook, Amazon, etc), we should encourage the national statistical institutions to start looking at big data that can help guide us through the myriad issues we are facing.

It is, of course, vital that these national institutions be based on democratic principles, and that they are not used for party political reasons. At present, the Trump government is looking at how they can use census data for the benefit of their own party politics. If this happens, we are one step closer to a very dangerous decline in our democracy.

Bureaus of Statistics were a result of the Enlightenment

Interestingly, many of our democratic institutions started their life in the 19th century as a consequence of the Enlightenment, when there was a new drive towards rational politics, scientific, social and economic developments. This needed to be underpinned by a framework of national measurements. The first National Bureau of Statistics was established in Paris in 1800. Over the last 200 years these institutions, which are now established in every country, have looked after uniformity in data collection, data integration, and data analytics, supported by a large group of independent and trusted data experts involved in interpreting the data that guided policy decisions for the benefit of all.

The effects of the Enlightenment have been enormous; and they are still being delivered. There are now more democratic countries than ever before; overall global poverty keeps decreasing; literacy keeps increasing; wars and the number of people killed in wars continue to decline; and average lifestyle around the globe keeps improving. We need to ensure that this upwards trend continues.

The philosopher Immanuel Kant, key person in what we might want to call the modern Enlightenment, described (in 1784) Enlightenment as follows:

‘Enlightenment is man’s emergence from his self-imposed immaturity. Immaturity is the inability to use one’s understanding without guidance from another.’

We most certainly have that ability to think for ourselves, and it is our responsibility as human beings to do so at a time when fake news, lies and other forces are trying to undermine our democratic values and principles. If we don’t stand up to these undermining forces they will cause extensive damage.

Decentralisation of facts and figures

A key problem now is that since many people no longer believe they are receiving positive social and economic outcomes they have ceased to trust the underlying data and are reverting to emotions, vague memories of a much better past, and imaginary futures.

While the statistical information that governments collected and used was certainly correct at macro-levels (levels of poverty, migration, GDP, unemployment, etc.), people live in micro-environments, and there the ‘facts and figures’ were quite different. And not just between national and local situations—importantly, there are great differences in facts and figures within towns, suburbs, rural regions and so on.

Differences within communities are significantly more complex than they were when these institutions were first established, and data collection began.

National governments unwilling to accept this level of criticism from their people will continue to lose trust. People don’t live in some artificial place as the national average—they live in real communities with real problems and issues which are not necessarily reflected in national facts and figures.

It has become clear that for our society to move forward, its governance needs to be more decentralised and that all of us need to participate in that process. And technology can assist us here.

Big data in connected cities

This also fits in with the understanding that cities need much better data to run their communities. There are great opportunities to win political trust back at these local levels. A key issue here is that this spatial decentralisation needs to be supported by functional decentralisation, so that cities, regions, and provinces have the autonomy enabling them to successfully address the local issues of education, healthcare, environment, jobs, economy, mobility, etc. Furthermore, a decentralisation of political systems and institutions is needed to assist these developments.

This does not mean that big data is not needed at a national level as well. It is equally essential there; but they, too, will need to decentralise. The National Bureaus of Statistics should be the nations’ leaders in big data for the common good, and they should not be used for party politics if they want to retain the position of trust that they currently still enjoy. But they will need to work far more closely with cities to better reflect the facts and figures of local communities.

The leading smart cities understand the need for these news structures. Councils of Mayors are becoming a new political force. Cities already have vast amounts of data that can be used to improve their local situations. However, to maximise the use of big data for the common good of their citizens a breakdown of the many (data) silos within their bureaucracies will be necessary. Little ivory towers where security, safety, and privacy issues are used to stop the data from being used in a broader and more open context.

An early lesson learned by smart city pioneers was that it is not about open slather data; it is about open data in a controlled environment.

Actively involving the local citizens in the various ‘smart city’ projects is critical and can generate further data relevant to their local situation. There are already some good examples in some of the leading smart cities.

Emotional and sentimental data

An emerging development is the taking of ‘emotional’ or ‘sentimental’ data analytics into account. We see this happening already in the commercial sphere (Facebook produced some interesting data reflecting emotional trends but was vilified for it, and as a result this form of data in now shrouded in secrecy—a very bad outcome indeed).

Another example is Cambridge Analytica—on whose board sits Steven Bannon. They developed psychological profiles for the Trump campaign and, again, great secrecy here also. Despite its potential little is happening so far in the public sphere in relation to the common good. Again, cities and communities could be a much better starting point for exploration of these softer data options, rather than the nation as a whole.

Big data is a far too important a development to be left just in the hands of commercial or ‘secret’ organisations. Cities that already have a holistic strategy in place could take a leadership role here. Within such a plan they will already have a data strategy in place and over time other cities and communities can learn from them and follow in their footsteps. Like trusted national statistical organisations, at a city level also we need professional statisticians and big data analysts who are able to make unambiguous and objective observations about their local economy and local community.