MGIE (MLLM-Guided Image Editing)

2024: Product Announcement

In February 2024, the open neural network MGIE (MLLM-Guided Image Editing) was launched, designed to edit photos for text requests. The technology was developed by Apple in conjunction with researchers at the University of California, Santa Barbara.

From the description for MGIE on GitHub, it follows that the development is multimodal, which is capable of working with several types of data. For example, a neural network can recognize natural language commands, images in the original photo, and generate new objects using a diffusion model. This approach allows you to combine several tasks in one neural network.

GitHub

MGIE can also replace the background of an image, add or remove objects, and apply "artistic effects" and color filters. With it, you can edit small details of the photo - face, hair, clothes, accessories.

When editing a photo with MGIE, users simply need to enter what they want to change in the image. The article gives an example of pepperoni pizza. 'Make it healthier'prompts prompted the AI to add more vegetable toppings to the photo. A picture of tigers in the Sahara looks dark, but after the models were told to "mimic more light," the image brightened.

MGIE is capable of improving the overall quality of photography, including brightness, contrast, clarity and color balance. She can apply artistic effects such as drawing, painting and caricatures.

The source code of the neural network is posted on GitHub. Users can test the capabilities of MGIE online via the Hugging Face Spaces platform (link), designed to collaborate on machine learning projects . It is possible that Apple will introduce the technology into its products.^[1]

Notes