TEXT TO VIDEO OPENS A NEW ERA OF AI
__________________________
John Fraim
The world of AI is changing so fast it is almost impossible to keep up with new aspects of the technology. In a general sense, one can make the observation that it started with text to text and progressed to text to images. The latest evolution of AI involves text to video.
One of the leading companies in creating text to video is OpenAI, developer of ChatGPT. OpenAI is the developer of Sora an AI model that can create realistic and imaginative scenes from text instructions. As noted on the AI site, “We’re teaching AI to understand and simulate the physical world in motion, with the goal of training models that help people solve problems that require real-world interaction.” As OpenAI says, “Sora can generate videos up to a minute long while maintaining visual quality and adherence to the user’s prompt.”
OpenAI notes on their site that “Sora is able to generate complex scenes with multiple characters, specific types of motion, and accurate details of the subject and background. The model understands not only what the user has asked for in the prompt, but also how those things exist in the physical world. The model has a deep understanding of language, enabling it to accurately interpret prompts and generate compelling characters that express vibrant emotions. Sora can also create multiple shots within a single generated video that accurately persist characters and visual style.”
* * *
As OpenAI says, “Today, Sora is becoming available to red teamers to assess critical areas for harms or risks. We are also granting access to a number of visual artists, designers, and filmmakers to gain feedback on how to advance the model to be most helpful for creative professionals. We’re sharing our research progress early to start working with and getting feedback from people outside of OpenAI and to give the public a sense of what AI capabilities are on the horizon.”
OpenAI realizes that Sora represents the beginning of a new era in images, video and filmmaking. One of the short films using Sora is “Air Head” (above) made by the Toronto multi-media company Shy Kids. Check out the comments on the video.
Open AI realizes there is a long way to go. They note, “The current model has weaknesses. It may struggle with accurately simulating the physics of a complex scene, and may not understand specific instances of cause and effect. For example, a person might take a bite out of a cookie, but afterward, the cookie may not have a bite mark. The model may also confuse spatial details of a prompt, for example, mixing up left and right, and may struggle with precise descriptions of events that take place over time, like following a specific camera trajectory.”
* * *
The use of AI for text to video is a milestone in the fast-developing world of AI. It’s an artificial world and this is a fascinating yet scary prospect for many. But this world is coming and Sora is on the forefront of it.
________________________________________
On Media Post, watch a video interview with Chief Creative Director Nik Kleverov of creative agency Native Foreign. Also, see some of the work he has done with Sora. Nik is one of a handful of those testing Sora now.