Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.
Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.
Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.
Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.
Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Thursday March 28, 2024 11:07 am PDT by Juli Clover
Apple will introduce new iPad Pro and iPad Air models in early May, according to Bloomberg's Mark Gurman. Gurman previously suggested the new iPads would come out in March, and then April, but the timeline has been pushed back once again. Subscribe to the MacRumors YouTube channel for more videos. Apple is working on updates to both the iPad Pro and iPad Air models. The iPad Pro models will...
In November, Apple announced that the iPhone would support the cross-platform messaging standard RCS (Rich Communication Services) in the Messages app starting "later" in 2024, and Google has now revealed a more narrow timeframe. In a since-deleted section of the revamped Google Messages web page, spotted by 9to5Google, Google said that Apple would be adopting RCS on the iPhone in the "fall...
Thieves in Montreal, Canada have been using Apple's AirTags to facilitate vehicle theft, according to a report from Vermont news sites WCAX and NBC5 (via 9to5Mac). Police officers in Burlington, Vermont have issued a warning about AirTags for drivers who recently visited Canada. Two Burlington residents found Apple AirTags in their vehicles after returning from trips to Montreal, and these...
Apple's WWDC 2024 dates have been announced, giving us timing for the unveiling of the company's next round of major operating system updates and likely some other announcements. This week also saw some disappointing news on the iPad front, with update timing for the iPad Pro and iPad Air pushed back from previous rumors. We did hear some new tidbits about what might be coming in iOS 18 and...
Photos of the first iPhone 16 cases have been shared online, offering another preview of the rumored new vertical rear camera arrangement on the standard iPhone 16 and iPhone 16 Plus. Image credit: Accessory leaker Sonny Dickson Over the last few months, Apple has been experimenting with different camera bump designs for the standard iPhone 16 models, all of which have featured a vertical ...
A $3 third-party app can now record spatial video on iPhone 15 Pro models in a higher resolution than Apple's very own Camera app. Thanks to an update first spotted by UploadVR, Spatialify can now record spatial videos with HDR in 1080p at 60fps or in 4K at 30fps. In comparison, Apple's native Camera app is limited to recording spatial video in 1080p at 30fps. Shortly after Apple's Vision ...
Top Rated Comments
Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.
Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.