Identifying Nepenthes species with AI

My plant website, Tom’s Carnivores, hosts a free tool for identifying tropical pitcher plants. You upload a photo, it finds every pitcher in the frame, and it tells you which species it thinks each one belongs to.

I can’t take any credit for the clever bit. The AI was built and trained by Eric Huang, an engineer and fellow grower in the US. My contribution was the front-end, and a home for it on my site. You can try the Nepenthes Species Identifier here.

A crash course in Nepenthes

Nepenthes are carnivorous plants from the tropics, mostly Southeast Asia. Their leaves end in pitchers: traps which lure insects in and digest them. There are nearly 200 species, and telling them apart is hard, because:

One plant makes very different pitchers. Young plants produce lower pitchers at ground level. Once they start climbing, they switch to upper pitchers, which can look like a different plant entirely (see below).
Species cross freely. Almost any two species can be crossed, and the offspring are fertile, so hybrids are everywhere.
No two plants are alike. Plants grown from seed vary a lot, even within a species.
Labels are often wrong. My own N. maxima was sold as N. eymae.

Two pitchers of Nepenthes lowii side by side. On the left, a squat, hairy, dark red lower pitcher with a ribbed rim and a bristly lid. On the right, a smooth green and maroon upper pitcher with a wide, flared mouth, hanging from a tendril in the forest. — Lower (left) and upper (right) pitchers of the same species, N. lowii.

For image recognition, that means lots of species which look alike, and individual species whose pitchers look nothing alike. The training data is messy too: popular species have plenty of photos, rare ones barely any. And since the AI only knows species, it has to give a species answer even when it’s shown a hybrid.

How it works

The identifier works in two steps:

Find the pitchers. The first model picks out every pitcher in the photo and traces its outline. Eric didn’t draw thousands of outlines by hand to train it; he used Meta’s Segment Anything model to draw them for him.
Name each one. Each pitcher is cut out and passed on its own to a second model, which returns its top five guesses. This is a vision transformer: the same basic technology behind ChatGPT, applied to images.

Both models run on an ordinary server, without a GPU, so photos are processed one at a time. My front-end uploads the user’s photo and checks every two seconds for the result, showing your place in the queue while a drawing a cool (but pointless) scan line. The result contains each pitcher’s top five guesses, plus its outline as a black-and-white mask. The front-end uses these to highlight the pitchers. When you tap on one, it works out which mask you’ve tapped, crops that pitcher out, and shows its predictions, each linked to my profile of that species (if I have one).

Architecture diagram. Users connect through Cloudflare to an Oracle Cloud server running Nginx, Gunicorn, Flask, Redis, PyTorch and SQLite. Below, the training pipeline: pitcher photos are labelled on UC San Diego's GPUs, then used to train the models on vast.ai. — How it all fits together. Diagram by Eric Huang.

For the curious, the models are PointRend and Swin Transformer V2, and Eric’s code is on GitHub.

Seeing it in action

Here’s a photo from my greenhouse. The AI has found four pitchers, and I’ve tapped on the N. singalana in my hand:

The identifier's results: my photo of a large dark pitcher held in my hand, with three other detected pitchers highlighted, beside the predictions N. singalana 94.9%, N. bongso 4.4% and N. izumiae 0.5%. — The identifier correctly names my N. singalana.

Correct, with 94.9% confidence. I often find the runners-up more interesting, though: N. bongso and N. izumiae both come from the same Sumatran mountains. The AI was never told where anything grows, but neighbouring species tend to be close relatives, and it picks up on the resemblance.

But hang on, what’s that half-formed (and vaguely obscene-looking) pitcher on the left?

The same photo with the small, unopened green pitcher on the left selected, and the predictions N. robcantleyi 92.0% and N. truncata 5.2%. — The same photo, with the unopened pitcher selected.

It’s a young pitcher on a different plant, still sealed shut, and the AI has correctly identified it as N. robcantleyi. It’s missing most of the features a grower would look for, and the AI can’t see the leaves or anything else outside the pitcher. I find this remarkable.

Finally, here’s what happens when I give it a hybrid, N. lowii x truncata:

The identifier's results for a red-striped hybrid pitcher, with the predictions N. truncata 78.6% and N. lowii 20.8%. — Identifying the parents of N. lowii x truncata.

It’s never been shown a hybrid, so it splits its confidence between the two species it recognises, which happen to be the parents. It doesn’t always work, but it often gets one or both.

Limitations and what’s next

The Nepenthes Species Identifier is a fun application of AI, not a scientific tool! It always gives an answer, even for a hybrid or a species it’s never seen, and high confidence doesn’t mean it’s right. It struggles with rare species and awkward photos. An early version had a suspicion that my cat was a Nepenthes ovata.

Proper hybrid identification is the obvious next step, and a hard one: there are thousands of possible crosses, and the offspring of a single cross can look wildly different from each other. We’d also like to let growers submit their own photos to help train future versions. If you have ideas, please get in touch via email.

Thanks for reading! Give the identifier a try, and let me know how it gets on with your plants.