How to Create Subject-Steady Animations with Dynamic Backgrounds

This tutorial illustrates how to craft an animation where the subject remains constant, while the background undergoes animation, and individual camera angle changes are applied to both elements.

1. Generate an image containing the subject you want to use. (Save it in the gallery.)

2. Generate an image containing the background you want to animate. (Save it in the gallery.)

3. Drag and drop the subject image into the ControlNet module.

4. Enable the CN Segmentation processor (2nd to last button on the left side of the image), and enable the subject mask button as well.

5. Make sure that the mask background color is set to "clear"

6. Select the ControlNet InPaint model.

7. Drag and drop the background image into the Guide Image module. Set the Strength value not higher than 40%. A high strength value will make the subject blend with the background. Tip: You can get away with higher strength values when using LCM models.

8. Click "Imagine" to create a single image and try out your settings.

9. Tweak the prompts, steps, guidance, guide image strength, and so on as needed. If you're using an LCM model, switch on "Real-time Generation" by clicking the lightning button at the top right of the Guidance knob. This lets you see how your adjustments look as you make them.

10. When you're happy with how the image blends the subject and background, go ahead and activate Animation from the top left toolbar.

11. Set the desired animation duration and Frames Per Second (FPS)

12. Disable Automatic Guide Image and Automatic ControlNet (1st button on the right side of each image)

13. Enable "Camera Zoom & Pan" by clicking the second button on the right side of each image

14. Set the initial position for each camera.

15. In the Keyframes module, add a keyframe at the end of the timeline.

16. With the last keyframe selected, set the final position for each camera.

17. Save the project.

18. Click "Dream" to generate the video.

19. Save the video to your Gallery.

Example subject-steady animation with dynamic background generated in Sogni:

Optional Cleanup Steps:

You might be satisfied with the video you generated earlier, but depending on the model you used and other settings, the animation might give the impression that the subject and background layers don't quite mesh together. This could be due to differences in lighting, contrast, and other factors. Or it might simply look a bit rough, especially if it was rapidly generated using LCM. Follow these steps to enhance the cohesion between the subject and background: 1. Close the previous project and start a new one.

2. (Optional) change the SD model.

3. Ensure that the animation duration and FPS (frames per second) settings match those used previously.

4. Drag the previously generated video from the Gallery and drop it into the Guide Image module.

5. Disable Animation, then click "Imagine" to generate a single image and test your settings. Make any necessary adjustments. Tip: You can utilize the Guide Image playback slider to experiment with different frames from the video.

6. When you're satisfied with your settings, activate Animation from the top left toolbar.

7. Click "Imagine" to re-generate the video.

Example Project File

About loading the example project files:

Be sure that you're using atleast Sogni v2.0.5, which includes an important bug fix for opening project files containing images with transparency. Included are project files that showcase all the settings used to generate the sample videos or serve as a starting point. There are two project files provided inside the .zip file: one for generating the initial video and another for the Cleanup steps. Project 1 file requirements:

• Download and load the LCM - 🅂 Sogni 𝛂1 model.

• If you haven't already, download the CN Segmentation preprocessor.

• If you haven't already, download the InPaint CN model.

• Open the project file "backgroundZoomSample_10s_LCM.sogni".

Project 2 file requirements:

• Download and load the SDXL-Turbo - 🅂 Sogni.XLT 𝛂1 model.

• Open the project file "backgroundZoomSample_10s_TurboCleanup.sogni".