19. April 2023
In a proof of concept we made a component for Adobe AEM that integrates a generative AI into the CMS to create images by using a textual description. We used Stable Diffusion which is available for use in a commercial setting without license cost. This saves time and money and ensures a consistent visual style. The aim was to integrate it seamlessly into the authors workflow to streamline content creation on the fly.
Background
Recent developments in AI has led to some remarkable pieces of software which can generate images from textual prompts. The main driver behind this are very large trained AI language models based on neural nets. There are several of these, notably Imagen (Google), Dall-E2 (OpenAI), Midjourney, Craiyon and Stable Diffusion (LMU Munich and Stability AI). The Stable Diffusion model was trained on 5 billion image-text pairs. Most of these are available in a limited form, either by restricting the access or license. But Stable Diffusion code and model weights have been released publicly and can be used in commercial setting without license cost.
Motivation / Use Case
We wanted to explore a use case which seamlessly integrates into the workflow of a content manager.
Valtech is an Adobe Partner and works a lot with the Adobe Experience Manager (AEM) Web CMS for client projects. So the most obvious thing to do was to create an AEM component which takes text and generates an image from it. To use the component, the author simply enters a description of the desired image and the CMS component will generate it. The image is then displayed on the webpage as usual. This can save time and effort in sourcing appropriate images for their content and can help ensure a consistent visual style across a website or other digital platform. Especially when it comes to decorative images that "just" accompany the actual text.
timelapse urban mobility at night, long-time exposure
Proof of concept
To play around, a proof-of-concept was created: we wanted to be as self sufficient as possible and used the Stable Diffusion software for creating the images. Stable Diffusion has a REST API, which runs on a newer Apple MacBook Pro (M1/M2 chip needed). Also the API can be purchased as SaaS from different providers.
Implementation
On the backend side, the stable diffusion REST API was set up on an M1 Mac, which returns an image by receiving a textual descripton.
On the AEM side, a CMS component was created where, the author inputs the description. The component then sends a request to the API which returns the generated image. The generated image is displayed in the CMS page, the text is used in its ALT-attribute.
Conclusion
The proof-of-concept is easy (and fun!) to use for the author and can generate useful license-free images. It integrates very well into Adobe AEM. The author can play around with the text to improve the images and style until it fits the content.
Future developments
We will publish the AEM component as open source software and enhance it to support different API providers.
Stable diffusion supports inpainting (predict missing image parts), outpainting (generate parts outside of an existing image) and other enhancements of existing images. We plan to test these and add them to the AEM component if it is worthwhile.
We also plan to adapt this component for other CMS and DXP partners which are used at Valtech such as Sitecore or Optimizely.
