# Inception v3 scores 62% to frontier LLMs' 42% on an LLM skin cancer benchmark

> A 2015 CNN fine-tuned on ISIC Archive images beat GPT-6 Astra and Claude Fable 5.1 on 200 biopsy-proven MILK10k lesions. The result bears on whether to train a classifier or call an LLM.

- Published: 2026-10-25
- By: Samir Sengupta (https://www.samcodeman.com/about)
- Canonical: https://www.samcodeman.com/writing/inception-v3-62-vs-frontier-llms-42-skin-cancer-benchmark
- How this was made: Drafted with Claude, Anthropic's model from 1 cited source, then checked against it in a second editing pass. Editorial policy: https://www.samcodeman.com/editorial-policy
- Disclosure: I build Tron-1B, an open-weight decision model, so I have a stake in how this category is judged.

## The short version

- On 200 biopsy-proven MILK10k lesions, a fine-tuned Inception v3 named the correct one of six diagnoses 125 times (62%). GPT-6 Astra managed 84 and Claude Fable 5.1 managed 85 (42%).
- At its preset threshold the CNN discharged 18 of 30 melanomas, against 3 for GPT-6 Astra and 4 for Claude Fable 5.1. With its threshold matched to the LLMs, it discharged 2 or 1.
- Each LLM leaned toward one favourite answer. GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were. Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were.
- The CNN was fine-tuned for about twenty minutes on a single H100 and runs in a phone browser. Commenters on Hacker News noted that the tradeoff is the labelled data and engineering a classifier needs.

In an LLM skin cancer benchmark published by dermatologist Magnus Lynch on October 25, 2026, a fine-tuned Inception v3 CNN beat GPT-6 Astra and Claude Fable 5.1 on the same images. The test set was 200 biopsy-proven lesions from the public MILK10k dataset. The CNN named the correct one of six diagnoses for 125 lesions (62%), against 84 and 85 (42%) for the two LLMs. Inception v3 was first published by Google in 2015, and this version was fine-tuned for about twenty minutes.

Lynch describes the CNN as deliberately not state of the art, and the test is small. It still matters to engineers deciding whether a vision task needs a trained classifier or can be handed to a frontier model through an API. On diagnosis accuracy the CNN wins clearly. On the question of which lesions can be safely discharged, the result reverses at the CNN's preset threshold: it discharged 18 of 30 melanomas there, against 3 and 4 for the LLMs.

## How the 200-lesion MILK10k test was built

MILK10k is a public dataset released through the International Skin Imaging Collaboration in 2025. It photographs each lesion twice, once as a clinical close-up and once as a dermoscopic image taken through the illuminated magnifying lens dermatologists use. Lynch drew 200 lesions at random, all confirmed by histopathology. The 100 that should not be discharged were 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas (including Bowen's disease) and 20 actinic keratoses. The 100 that are safe to discharge were 50 benign naevi and 50 seborrhoeic keratoses.

Every lesion was biopsied, including the benign ones. The author says this makes them harder to diagnose, because a mole that somebody decided to cut out is one that worried a clinician. The LLMs were given both images and nothing else, with the same prompt at default settings. For each lesion they were asked for:

- the most likely of the six diagnoses, or unsure
- a confidence
- a second choice
- whether the lesion could be safely discharged with no treatment, biopsy or follow-up

Neither model refused a case. The CNN was Inception v3 pretrained on everyday photographs, then fine-tuned on all layers using 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU. The 200 test lesions, and every other photograph of the same lesion or patient, were removed from its training and validation data. They were also not used to choose the model or set its threshold. The CNN scored both images, put most of the weight on the dermoscopic one, and chose from the same six diagnoses.

## Where the 62% versus 42% gap comes from

With a second choice allowed, the CNN reached 173 of 200, against 134 for GPT-6 Astra and 138 for Claude Fable 5.1. On melanoma as first choice it named 13 of 30, against 8 and 11. The biggest gap was on the harmless lesions, while the LLMs held their own on two of the cancers:

- Moles (of 50): CNN 37, GPT-6 Astra 16, Claude Fable 5.1 33.
- Seborrhoeic keratoses (of 50): CNN 26, GPT-6 Astra 13, Claude Fable 5.1 6.
- Basal cell carcinoma: the CNN was one short of GPT-6 Astra.
- Actinic keratosis: the CNN was behind both LLMs.

The models also failed differently. Each LLM had a favourite answer. GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas. Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were. The CNN's melanoma errors went toward benign labels: of the 17 it got wrong, it called 8 a naevus and 6 a seborrhoeic keratosis.

The source does not test whether reordering the answer options or rephrasing the prompt changes the LLMs' biases. One commenter on Hacker News said LLM decision models are sensitive to both, and that they start to break down beyond about 20 classes.

## Does the CNN discharge more melanomas?

At its preset threshold, yes. The CNN kept 77 of 100 unsafe lesions and discharged 18 of 30 melanomas. GPT-6 Astra kept 97 and discharged 3 melanomas, and Claude Fable 5.1 kept 96 and discharged 4. The CNN also discharged far more of the safe lesions: 76 of 100, against 32 and 22.

When its threshold was matched to the LLMs, the CNN kept 98 or 99 unsafe lesions and discharged 2 or 1 melanomas. The source does not say how many safe lesions it discharged at that matched threshold, so the full tradeoff at equal sensitivity is not published. The CNN's area under the ROC curve for separating lesions that should and should not be discharged was 0.86 on these 200 lesions and 0.96 on ordinary archive images. No AUC is reported for the LLMs.

For engineers, the difference is in what each system returns. The classifier returns a score, and its operating point can be moved. The LLMs were asked for a confidence, but the source does not report thresholding on it, so their discharge behaviour comes from their verbal decision. The LLMs were cautious on discharge while naming the correct diagnosis for only 42% of lesions.

## What 200 lesions and one run cannot show

One commenter on Hacker News, writing as an ML engineer, dismissed the test as a fine-tuned CNN compared against general purpose models on a tiny dataset. The author makes a similar caveat. Two hundred lesions show the pattern but cannot rank close results, and 30 melanomas leave wide uncertainty on the figure that matters most. Each LLM case was run once and answers vary between runs, so the source gives no estimate of run-to-run variance.

Contamination would have favoured the LLMs. MILK10k has been public since 2025, so the LLMs could have seen it in training, while the CNN certainly had not. The author also notes that state-of-the-art skin cancer AI uses ensembles of image transformers trained on more data, and that regulated products built on them perform considerably better. The analysis and first drafts were produced with Claude Code, and one of the two tested models is also Anthropic's. Scoring was done by script against the histology.

One condition was run as a patient would ask. The LLMs saw only the clinical photograph and the question "do I need to see a doctor?" Both advised seeing a doctor in almost every case, harmless lesions included, for 99 and 98 of a further 100 biopsy-proven melanomas.

## Tool calls are the untested middle path

One commenter on Hacker News compared the result to a calculator beating an LLM at arithmetic. The point was that once the LLM can call the calculator, the calculator's advantage disappears. The equivalent here would be an LLM that routes the image to a classifier like this Inception v3 and reasons over its scores. The source does not test that configuration, so there is no number for it. The CNN is small enough to run entirely in a phone browser using WebGPU with nothing uploaded, so it could in principle be called locally as a tool.

## When a small classifier beats a general LLM

The usual objection to a custom classifier is the cost of building it. Commenters on Hacker News said an LLM needs only a prompt, while a classifier needs labelled training data and engineering effort. In this case much of that cost had already been paid. There were 68,011 labelled ISIC Archive images available, and fine-tuning took about twenty minutes on one H100. Under those conditions, the specialised model won on first-choice accuracy by about 20 percentage points.

- Use a small classifier when public or internal labelled data exists for a fixed label set (six classes here). It is also the better fit when you need to set and audit a threshold, or need on-device inference.
- Use a general LLM when no labelled data exists, the task is open-ended, and good-enough accuracy is acceptable. Commenters on Hacker News pointed out that skin cancer is probably not such a task.
- One commenter suggested a third route: use the LLM in production for the cases where it is most confident, then train the specialised model on the collected usage.
- Whichever you choose, compare models at matched operating points. In this test, the CNN at its preset threshold discharged 18 of 30 melanomas.

The cases, prompts, every LLM reply and the scoring code are published in one download. Teams can use them to rerun the comparison with multiple LLM runs or a tool-calling setup. The source reports neither LLM latency nor per-case API cost.

## Questions this raises

### Did the fine-tuned CNN beat frontier LLMs at diagnosing skin lesions?

Yes, on diagnosis accuracy. Inception v3 named the correct diagnosis for 62% of 200 biopsy-proven MILK10k lesions, while GPT-6 Astra and Claude Fable 5.1 both scored 42%. The biggest gap was on harmless moles and seborrhoeic keratoses.

### Is the CNN safer than the LLMs at deciding which lesions can be discharged?

Not at its preset threshold, where it discharged 18 of 30 melanomas against 3 and 4 for the LLMs. When its threshold was matched to the LLMs it discharged only 1 or 2 melanomas, but the source does not report how many safe lesions it discharged at that setting.

### How reliable is a 200-lesion test like this?

The author says 200 lesions show the pattern but cannot rank close results, and 30 melanomas leave wide uncertainty. Each LLM case was run only once, so there is no estimate of run-to-run variance.

## Key facts

- Fine-tuned Inception v3 named the correct one of six diagnoses for 125 of 200 MILK10k lesions (62%), vs 84 and 85 (42%) for GPT-6 Astra and Claude Fable 5.1.
- With a second choice allowed, the CNN reached 173 of 200, against 134 for GPT-6 Astra and 138 for Claude Fable 5.1.
- At its preset threshold the CNN discharged 18 of 30 melanomas, against 3 for GPT-6 Astra and 4 for Claude Fable 5.1.
- The CNN was fine-tuned on all layers using 68,011 ISIC Archive images on a single H100 GPU for about twenty minutes.
- The CNN's AUC for discharge decisions was 0.86 on the 200 test lesions and 0.96 on ordinary archive images. No AUC is reported for the LLMs.

## Sources

1. [A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis](https://lynchlab.net/llm-skin-cancer) - lynchlab.net

Discussion read (practitioners' reactions, not a source): https://news.ycombinator.com/item?id=50233779