Adding custom gestures to an AR application with SwiftUI

In the article Using RealityKit gestures in an AR application with SwiftUI we saw how we can easily integrate gestures that RealityKit provides out of the box. In addition to this type of gesture, there are some others that come from the UIGestureRecognizerit class.

Implementing those gestures on a RealityKit entity is a bit more challenging because we need some UIKit functionality. Let’s dive into creating a long-press gesture that will delete the 3D model from the scene.

The UIGestureRecognizer class

The UIGestureRecognizer class has the responsibility of detecting touch patterns on the screen. It offers various subclasses designed for recognizing specific patterns, making it possible to implement different types of gestures in apps. You can find the complete list of subclasses and how the UIGestureRecognizer works in the Apple Documentation.

UIGestureRecognizer | Apple Developer Documentation

The base class for concrete gesture recognizers.

Apple Developer Documentation

Connect SwiftUI to UIKit through the Coordinator

As said before UIGestureRecognizer comes from UIKit, but our app is written in SwiftUI. What we need is a bridge to connect these two frameworks. This bridge is called Coordinator, primarily used to manage the integration of UIKit components or functionality into a SwiftUI-based app.

In the UIViewRepresentable, where we set the ARView session additionally to the makeUIView and updateUIViewmethod, create the Coordinator class.

In the Coordinator we declare an optional variable of type ARView. We will need that to access the AR session.

struct ARViewContainer: UIViewRepresentable {

  func makeUIView(context: Context) -> ARView {
    ...
  }

  func updateUIView(_ uiView: ARView, context: Context) {
    ...
  }

  // Creating the Coordinator class
  class Coordinator: NSObject {
    var view: ARView?
  }

}

Within this class, specify the method that defines what happens when a gesture is recognized. In our example, we remove the 3D model when the long press gesture is performed using the UITapGestureRecognizer subclass.

class Coordinator: NSObject {
    var view: ARView?
        
    @objc
    func handleLongPress(_ recognizer: UITapGestureRecognizer? = nil) {
        // Check if there is a view to work with
        guard let view = self.view else { return }

        // Obtain the location of a tap or touch gesture
        let tapLocation = recognizer!.location(in: view)

        // Checking if there's an entity at the tapped location within the view
        if let entity = view.entity(at: tapLocation) as? ModelEntity {
  
            // Check if this entity is anchored to an anchor
                if let anchorEntity = entity.anchor {
                    // Remove the model from the scene
                    anchorEntity.removeFromParent()
                }
        }
    }
  
}

The Coordinator class is ready. We need to create a way to create a coordinator object for our SwiftUI view to work with. The makeCoordinator() method is responsible for initializing and providing a coordinator instance. Inside the UIViewRepresentable structure.

struct ARViewContainer: UIViewRepresentable {

    // Instantiate and return a Coordinator object
    func makeCoordinator() -> Coordinator {
        return Coordinator()
    }

    func makeUIView(context: Context) -> ARView { ... }

    func updateUIView(context: Context) -> ARView { ... }

    class Coordinator: NSObject { ... }

}

Add the gesture recognizer on the ARView

The last step is to add the gesture recognizer to the ARView session. This is a complete example of the updateUIView method in which we apply the gesture in the augmented reality session.

Here is what the new implementation of the updateUIView(_:context:) method should look like.

func updateUIView(_ uiView: ARView, context: Context) {
    let anchorEntity = AnchorEntity(plane: .any)
    
    guard let modelEntity = try? Entity.loadModel(named: modelName) else { return }

    modelEntity.generateCollisionShapes(recursive: true)
    
    anchorEntity.addChild(modelEntity)

    // Set the 'view' property of the coordinator to the 'uiView' passed as an argument.
    context.coordinator.view = uiView

    // Create a UILongPressGestureRecognizer to detect long-press gestures.
    let longPressGesture = UILongPressGestureRecognizer(target: context.coordinator, action: #selector(Coordinator.handleLongPress(_:)))

    // Add the UILongPressGestureRecognizer to the 'uiView' for user interaction.
    uiView.addGestureRecognizer(longPressGesture)

    uiView.scene.addAnchor(anchorEntity)
}

Conclusion

Let’s see how we can interact with our 3D model through the specific gesture we've defined. 

Custom gestures create a unique interaction that users can perform with digital objects in augmented reality. There are other resources that you can check if you want to go deeper into augmented reality apps.

• Creating an augmented reality app in SwiftUI using RealityKit and ARKit
• Using the ARKit coaching view in a SwiftUI AR application
• Using RealityKit gestures in an AR application with SwiftUI