# Identifying Nepenthes species with AI

> How the Nepenthes Species Identifier finds every pitcher in a photo and names its species, and why pitcher plants make that surprisingly hard.

By [Tom Bennet](https://tombennet.dev/) · 4 Sep 2024 · 4 min read

My plant website, [Tom's Carnivores](https://tomscarnivores.com), hosts a free tool for identifying tropical pitcher plants. You upload a photo, it finds every pitcher in the frame, and it tells you which species it thinks each one belongs to.

I can't take any credit for the clever bit. The AI was built and trained by **Eric Huang**, an engineer and fellow grower in the US. My contribution was the front-end, and a home for it on my site. You can [try the Nepenthes Species Identifier here](https://tomscarnivores.com/resources/nepenthes-identifier/).

## A crash course in Nepenthes

_Nepenthes_ are carnivorous plants from the tropics, mostly Southeast Asia. Their leaves end in pitchers: traps which lure insects in and digest them. There are nearly 200 species, and telling them apart is hard, because:

- **One plant makes very different pitchers.** Young plants produce _lower_ pitchers at ground level. Once they start climbing, they switch to _upper_ pitchers, which can look like a different plant entirely (see below).
- **Species cross freely.** Almost any two species can be crossed, and the offspring are fertile, so hybrids are everywhere.
- **No two plants are alike.** Plants grown from seed vary a lot, even within a species.
- **Labels are often wrong.** My own _N. maxima_ was sold as _N. eymae_.

![Two pitchers of Nepenthes lowii side by side. On the left, a squat, hairy, dark red lower pitcher with a ribbed rim and a bristly lid. On the right, a smooth green and maroon upper pitcher with a wide, flared mouth, hanging from a tendril in the forest.](https://tombennet.dev/_astro/nepenthes-lowii-lower-upper.D4XmEYVi.webp)

Lower (left) and upper (right) pitchers of the same species, N. lowii.

For image recognition, that means lots of species which look alike, and individual species whose pitchers look nothing alike. The training data is messy too: popular species have plenty of photos, rare ones barely any. And since the AI only knows species, it has to give a species answer even when it's shown a hybrid.

## How it works

The identifier works in two steps:

1. **Find the pitchers.** The first model picks out every pitcher in the photo and traces its outline. Eric didn't draw thousands of outlines by hand to train it; he used Meta's [Segment Anything](https://segment-anything.com/) model to draw them for him.
2. **Name each one.** Each pitcher is cut out and passed on its own to a second model, which returns its top five guesses. This is a _vision transformer_: the same basic technology behind ChatGPT, applied to images.

Both models run on an ordinary server, without a GPU, so photos are processed one at a time. My front-end uploads the user's photo and checks every two seconds for the result, showing your place in the queue while a drawing a cool (but pointless) scan line. The result contains each pitcher's top five guesses, plus its outline as a black-and-white mask. The front-end uses these to highlight the pitchers. When you tap on one, it works out which mask you've tapped, crops that pitcher out, and shows its predictions, each linked to my profile of that species (if I have one).

![Architecture diagram. Users connect through Cloudflare to an Oracle Cloud server running Nginx, Gunicorn, Flask, Redis, PyTorch and SQLite. Below, the training pipeline: pitcher photos are labelled on UC San Diego's GPUs, then used to train the models on vast.ai.](https://tombennet.dev/_astro/nepenthes-identifier-architecture.dKfK15Vu.webp)

How it all fits together. Diagram by Eric Huang.

For the curious, the models are [PointRend](https://arxiv.org/abs/1912.08193) and [Swin Transformer V2](https://arxiv.org/abs/2111.09883), and Eric's [code is on GitHub](https://github.com/praxxus11/SegmentationService).

## Seeing it in action

Here's a photo from my greenhouse. The AI has found four pitchers, and I've tapped on the _N. singalana_ in my hand:

![The identifier's results: my photo of a large dark pitcher held in my hand, with three other detected pitchers highlighted, beside the predictions N. singalana 94.9%, N. bongso 4.4% and N. izumiae 0.5%.](https://tombennet.dev/_astro/ai-identifies-n-singalana.CZZpIKLJ.webp)

The identifier correctly names my N. singalana.

Correct, with 94.9% confidence. I often find the runners-up more interesting, though: _N. bongso_ and _N. izumiae_ both come from the same Sumatran mountains. The AI was never told where anything grows, but neighbouring species tend to be close relatives, and it picks up on the resemblance.

But hang on, what's that half-formed (and vaguely obscene-looking) pitcher on the left?

![The same photo with the small, unopened green pitcher on the left selected, and the predictions N. robcantleyi 92.0% and N. truncata 5.2%.](https://tombennet.dev/_astro/identification-n-robcantleyi-partial-pitcher.vvEu496N.webp)

The same photo, with the unopened pitcher selected.

It's a young pitcher on a different plant, still sealed shut, and the AI has correctly identified it as _N. robcantleyi_. It's missing most of the features a grower would look for, and the AI can't see the leaves or anything else outside the pitcher. I find this remarkable.

Finally, here's what happens when I give it a hybrid, _N. lowii x truncata_:

![The identifier's results for a red-striped hybrid pitcher, with the predictions N. truncata 78.6% and N. lowii 20.8%.](https://tombennet.dev/_astro/identification-n-lowii-x-truncata-parents.N1s9s7c4.webp)

Identifying the parents of N. lowii x truncata.

It's never been shown a hybrid, so it splits its confidence between the two species it recognises, which happen to be the parents. It doesn't always work, but it often gets one or both.

## Limitations and what's next

**The Nepenthes Species Identifier is a fun application of AI, not a scientific tool!** It always gives an answer, even for a hybrid or a species it's never seen, and high confidence doesn't mean it's right. It struggles with rare species and awkward photos. An early version had a suspicion that my cat was a _Nepenthes ovata_.

Proper hybrid identification is the obvious next step, and a hard one: there are thousands of possible crosses, and the offspring of a single cross can look wildly different from each other. We'd also like to let growers submit their own photos to help train future versions. If you have ideas, please get in touch [via email](mailto:hello@tombennet.dev).

**Thanks for reading!** Give the [identifier](https://tomscarnivores.com/resources/nepenthes-identifier/) a try, and let me know how it gets on with your plants.