If my doctor uses an AI tool to assess my stroke risk, how do I know it's reliable?

It is a reasonable question to ask your GP directly. Legitimate, well-validated clinical AI tools will typically have published validation data in populations similar to yours, and your GP should be able to tell you which tool they are using and what it is based on. The concern raised by this study is specifically about AI prediction models built on datasets with no verifiable source — meaning clinicians cannot audit the data quality even if they want to. If you are uncertain, you can ask: 'Has this tool been validated in Australian patients?' and 'Where was the training data sourced from?' You are entitled to understand the basis of any clinical recommendation you receive.

Does this mean I should distrust AI in my GP's consulting room?

Not as a blanket position. The study identifies a specific and fixable problem — a lack of data provenance documentation — rather than demonstrating that AI tools are uniformly unreliable. Some clinical AI applications have strong evidence bases and have been validated in diverse populations. The concern is that clinicians and patients often cannot tell, from a published paper or a software product description, whether the underlying training data meets quality standards. The researchers' call for mandatory data transparency is the mechanism that would allow meaningful evaluation. In the meantime, asking questions and using AI-generated suggestions as one input rather than a final answer reflects sound clinical practice.

AI diagnostic tools for stroke and diabetes: shaky foundations

What just happened

A study from researchers at Queensland University of Technology, published in BMC Medicine, has found that 125 peer-reviewed clinical AI prediction models — covering stroke risk and type 2 diabetes — were built using data sourced from Kaggle, a public repository hosting over half a million datasets, where two specific datasets had no documentation of when, where, why, or how the data was collected.

The lead researcher, Alexander Gibson, and colleagues applied the TRIPOD+AI framework — an internationally recognised standard for evaluating clinical AI studies — to the datasets in question and found zero provenance documentation. The publications describing these models contained no information about the source population, data collection methodology, or quality controls.

Three of the 125 models identified in the study were in active clinical use. One appeared in a medical device patent. Two stroke risk calculators built on these datasets remain publicly accessible online. Practical clinical recommendations were present in 67% of the stroke articles and 80% of the diabetes prediction articles reviewed.

The both-and

AI in clinical decision support is not uniformly good or bad

There is a well-understood case for AI clinical prediction tools: they can synthesise complex combinations of risk factors faster than manual calculation, apply consistent weighting across variables that human clinicians may handle differently in different consultations, and surface risk patterns that are genuinely hard to hold in working memory during a busy appointment.

Several validated AI tools in cardiovascular and metabolic medicine have performed well against established risk scores and have been tested in populations representative of the settings where they are deployed. The clinical AI landscape is not a wasteland.

The problem this study identifies is not that AI tools are unreliable in general — it is that the quality of the training data cannot currently be assessed from outside, and that the field has no enforced standard requiring it to be disclosed.

Prediction is only as good as the population it was trained on

The TRIPOD+AI framework finding of zero provenance documentation is the central concern. A clinical prediction model trained on a dataset with unknown demographics, unknown data collection methods, and unknown quality controls may perform well in the population it was extracted from — and very badly in a different one.

Gibson et al. put the patient safety implication directly: “Data that is not fit for purpose can place patients at increased risk of harm as poor predictions will lead to patients being denied necessary treatments or receiving unnecessary treatments.”

This is not a hypothetical risk. A stroke risk calculator that was trained predominantly on one ethnic group may underestimate risk in another. A diabetes prediction model derived from a population with a different baseline rate of pre-existing conditions may miscalibrate the threshold at which it flags concern. These biases do not announce themselves; they operate quietly inside the model’s outputs.

The specific problem with Kaggle as a data source is not that the platform is inherently problematic — open data repositories have legitimate research uses. The problem is that the two datasets identified in this study were treated as if they were validated clinical datasets when neither had the documentation to support that status. Researchers in 32 countries built and published models on them, and the peer-review process did not catch the gap.

GPs are at a specific disadvantage

GPs currently have limited capacity to audit the AI tools presented to them, whether through clinical software integrations, electronic health record add-ons, or digital health platforms. The time required to evaluate the provenance and validation of an AI-generated risk score during a 15-minute consultation is not available.

This places an asymmetric burden on clinicians: they are expected to exercise professional judgement about AI-generated recommendations, but they cannot access the information that would allow them to exercise it meaningfully. An AI score does not currently come with a label that says “trained on an Australian population, validated in a real-world clinical setting, audited for demographic bias.” It comes with a number.

The researchers’ recommendation — mandatory data availability reporting as a condition of publication, and pre-registration of AI clinical studies — addresses the structural gap. It would not fix models already in circulation, but it would mean that future publications carry enough information for evaluation.

What anchoring looks like when it’s invisible

There is an additional concern worth naming. Clinicians reviewing AI-generated outputs in the course of a consultation are subject to the same anchoring cognitive bias as in any clinical scenario: a suggested risk score or differential, once seen, influences subsequent reasoning even when the clinician consciously reviews and corrects it. When the AI output is embedded in clinical workflow software rather than displayed as an explicit suggestion, that influence is further removed from scrutiny.

This is not unique to AI — paper-based risk scores carry the same anchoring risk. What differs is scale and opacity: when a poorly-founded AI model is integrated into a widely-used platform, its influence on clinical decision-making is population-level. The traditional safeguard — the peer-reviewed evidence base behind a paper risk score — is precisely what this study found to be absent.

2 cents

This story is worth knowing about for two reasons. First, it surfaces a concrete, addressable problem — data provenance disclosure — rather than a diffuse warning about AI being dangerous. The researchers know what would fix it. Second, it has direct patient implications right now: two of the stroke risk calculators flagged in the study remain publicly accessible online.

If you have used an online stroke risk tool in the last few years, the tool itself does not guarantee anything about the quality of its underlying model. A result from a non-validated tool is not necessarily wrong — but it is not independently meaningful either. Any clinical concern about stroke or cardiovascular risk is worth discussing with your GP directly, not managing via a consumer-facing web calculator.

For GPs, the question of which AI-generated recommendations to trust — and on what basis — is a genuine and underacknowledged clinical governance issue. Asking vendors about validation data is reasonable and, as tools integrate more deeply into practice software, increasingly necessary.

Verdict: maybe — a structural problem that warrants attention, but no immediate action required for most patients; the watch-this point is how the field responds to calls for mandatory data disclosure.

Sources cited

Medical Republic — AI stroke, diabetes tools built on dodgy datasets (10 July 2026). https://www.medicalrepublic.com.au/ai-stroke-diabetes-tools-built-on-dodgy-datasets/127226
Gibson A et al., BMC Medicine — Clinical prediction models trained on datasets with unknown provenance. https://doi.org/10.1186/s12916-026-04981-y