The Risks Are Big
On a broad level, a reactive approach to data and AI means the technology may not be used to its full potential. Moreover, poor quality data or data shortages can cause more direct problems.
If the datasets used to train and test AI systems are incomplete or of poor quality, the resulting algorithms will be biased. There have already been allegations that AI, if not properly trained or managed, can make discriminatory decisions based on characteristics such as gender or ethnicity.
For example, if an AI system trained on predominantly male data assesses credit applications, it may favor a man’s application over a woman’s due to underrepresentation of women in the dataset. This highlights the critical need for organizations to train AI models with diverse datasets and implement comprehensive reporting processes to identify and rectify any data biases.
Authorities at the highest level are also recognizing the implications of poor data quality. In the U.S., initiatives are underway to mandate that financial institutions and other organizations ensure “systems should be used and designed in an equitable way”. The EU’s upcoming AI Act and the UK government’s AI White Paper, A Pro-Innovation Approach to AI Regulation, includes similar provisions, as authorities worldwide seek to balance AI-driven innovation with regulatory oversight. Organizations that fail to meet these standards risk reputational damage and hefty fines, which in the EU could amount to up to 7% of annual worldwide turnover.1
Given the vast amount of data usage by financial institutions and the complexity of AI systems, those without in-house expertise or robust external AI support may become overly reliant on their technology. This dependency might lead them to accept whatever outcomes their systems produce, potentially overlooking inaccurate or discriminatory decisions as they do not have the ability or know-how to challenge them.
Understanding Technology and Data
To avoid these risks, organizations must fully understand the technology they deploy and the data utilized by these machines. They must ensure their models are validated before any AI system goes live, while generative AI must be carefully trained on diverse data and properly configured. Once live, it is equally important for organizations to regularly test systems to identify any flaws or deviations from expected performance. Where there are clear signs that datasets are inaccurate, organizations should promptly refresh and retrain their models.
