Apple quietly open-sourced a multimodal AI called Ferret

Ferret is an open-source multimodal large language model from Apple and Cornell University that lets you query specific, freeform parts of an image. Instead of analyzing an entire picture or relying on rigid bounding boxes, you can draw any shape on an image and ask questions about just that slice.

What it does

Standard multimodal models often struggle to isolate tiny or irregularly shaped details. Ferret tackles this by letting users sketch or highlight any freeform region on a picture.

According to Apple AI/ML research scientist Zhe Gan, Ferret can refer and ground anything anywhere at any granularity. It looks at the designated area, pinpoints key elements, draws an accurate bounding box, and seamlessly uses that item as part of your prompt.

For example, if you highlight one specific animal in a crowded picture and ask what it is, Ferret can identify the exact species, recognize that you are pointing to a single creature out of a group, and bring in surrounding context to give a full answer.

In terms of fine-grained precision, the researchers noted that Ferret often demonstrates better understanding of small image regions than GPT-4V.

Specs and availability

Apple originally pushed the code and the Ferret-Bench benchmark to GitHub on October 30 without any announcement, following up with model weight checkpoints on December 14.

The project caught fresh attention in December after AI figures like Bart De Witte highlighted it on X as a testament to Apple's AI commitments. Community members on Reddit also spotted that the model was trained on eight Nvidia A100 GPUs with 80GB of memory, marking a rare public nod from Apple to Nvidia hardware.

Researchers can inspect the code and download model checkpoints directly on GitHub today. However, the release is distributed under a non-commercial license, which means you cannot build commercial products with it right now.

Why it matters

If you are a multimodal researcher or computer vision engineer, Ferret is definitely worth spinning up. Its ability to parse arbitrary sketch inputs solves a real usability headache in image-based grounding.

We think this open-source move is a smart tactic for Apple. While its proprietary AI infrastructure continues to scale behind closed doors, shipping open weights earns goodwill and real feedback from the research community.

That said, the non-commercial license is a clear barrier for indie developers and startups. For now, look at Ferret as a high-performing reference model and a preview of the vision features Apple could eventually bring to its own hardware ecosystem.