Sony Announces Spatial Content Creation System, Equipped with High-Quality XR Head-Mounted Display and Controllers Dedicated to Interaction with 3D Objects

Sony Announces Spatial Content Creation System, Equipped with High-Quality XR Head-Mounted Display and Controllers Dedicated to Interaction with 3D Objects.

Metaverse Technologies and Hardware to Empower 3D Content Creators

TOKYO, January 9, 2024 -Sony Corporation announced development of an immersive spatial content creation system, which includes an XR head-mounted display equipped with high-quality 4K OLED Microdisplays*2 and video see-through function, and a pair of controllers optimized for intuitive interaction with 3D objects and precise pointing. The system is aimed at supporting creators in sophisticated 3D content creation. Sony plans to collaborate with developers of a variety of 3D production software, including in the entertainment and industrial design fields. At launch, Sony is exclusively partnering with Siemens, a leader in industrial technology, to introduce a new solution for immersive design and collaborative product engineering using software from the Siemens Xcelerator open digital business platform.

The system will be available later in 2024. Sony and its partners will announce further details including specs, launch date and regions, pricing, sales channels, and software compatibility.

This spatial content creation system, equipped with an XR head-mounted display with 4K OLED Microdisplays and Sony's proprietary rendering technology, enables real-time, high-definition and realistic rendering of textures of 3D objects and facial expressions of human characters. In addition to video see-through functionality and spatial recognition with six cameras and sensors in total, the system features a ring controller that allows users to intuitively manipulate objects in virtual space, as well as a pointing controller that enables precise pointing, allowing creators to craft in virtual space with controllers and keyboards, all while wearing the head-mounted display. Creators will not only be able to see real-scale 3D models in an Extended Reality (XR) environment with the high-definition display, but also create and modify 3D models in it. Through integration with third-party 3D creation applications, it also supports remote, real-time review between locations, making this an intuitive and immersive tool for the entire process of spatial content creation.

The device features Snapdragon® XR2+ Gen 2 Platform, the latest XR processor by Qualcomm Technologies, Inc. The processor unlocks the power of the 4K OLED Microdisplays to deliver stunning image quality, as well as provides user and space tracking for seamless XR experiences, offering creators a high-performance platform for their creative workflow.

Sony has released technologies that support workflow and quality of spatial content creation and facilitate use of 3D CG in various creative fields. Mobile motion capture system "mocopi™," with small and lightweight sensors and a dedicated smartphone app, enables full-body motion tracking, using Sony's proprietary technologies. Spatial Reality Displays provide highly realistic, three-dimensional content without the use of special glasses or Virtual Reality (VR) headsets, for spatial content creation and review process. By introducing this system and collaborating with 3D creation software developers, Sony aims to further empower spatial content creators to transcend boundaries between the physical and virtual realms for more immersive creative experiences.

"Siemens is partnering with Sony to enable immersive engineering, a critical building block for the industrial metaverse. Together, we are creating an environment where it will be possible to experience the realities of physics, without the bounds of time to profoundly improve how our customers work and collaborate," said Cedrik Neike, Member of the Managing Board of Siemens AG and CEO of Siemens Digital Industries.

"We designed our Snapdragon XR2+ Gen 2 to deliver 4K display resolution, cutting-edge graphics, and unparalleled performance to enable creators to build innovative spatial content that will transform all industries for the better," said Said Bakadir, Senior Director, Product Management of Qualcomm
Technologies, Inc. "It's exciting to see Sony enter the enterprise space with their spatial content creation system and utilize Snapdragon XR technologies to unleash more realistic, detailed, and precise Mixed Reality (MR)/VR experiences that will help developers and creators push forward an even more immersive future."

Main Features

High-Quality XR Experience for Creative Uses

The head-mounted display is equipped with large-size, high-definition 1.3-type OLED Microdisplays with 4K resolution and a wide color gamut that covers up to 96% of DCI-P3*3*4. Detailed and realistic rendering of textures of 3D objects and facial expressions of human characters allows creators to go through the whole process of modeling to review, all while wearing the head-mounted display.

By supporting split rendering*5, which distributes the rendering load between computers and the head-mounted display, the system is capable of stable and high-definition rendering of large-size 3D models.

Through our collaboration with Qualcomm Technologies, Inc., Sony is among the first to adopt their latest Snapdragon XR2+ Gen 2 platform. The processor unlocks the power of the 4K OLED Microdisplays to deliver stunning image quality, while on-chip AI enables enhanced user and space tracking for seamless XR experiences, merging physical and digital spaces for easy navigation and intuitive interaction within the XR creative spaces.

Controllers that Provide Intuitive Spatial Content Creation Experiences

The system includes a ring controller that allows users to intuitively manipulate objects in virtual space. It also comes with a pointing controller that enables stable and accurate pointing in virtual spaces, with optimized shape and button layouts for efficient and precise operation. By holding the pointing controller in the dominant hand and attaching the ring controller to the fingers of the other hand, creators can model 3D objects using both controllers and a keyboard, while wearing the head-mounted display.

Intuitive manipulation of an object with the ring controller

The pointing controller enables
accurate pointing in virtual spaces

Designed with Comfort, for Seamless XR Creative Experiences

The head-mounted display provides comfort and stability required for extended creative use, by fine-tuning the balance of the device's center of gravity, and the design and material for pads that come into direct contact with the user's head. Additionally, the flip-up mechanism of the display part allows creators to easily and seamlessly move between physical and virtual spaces, without having to recalibrate the device when removing and wearing the entire device.

Optimal balance for extended creative uses

Flip-up mechanism of display part

Immersive Creative Experience Through Integration with 3D Creation Software

Sony seeks to collaborate with various 3D production software developers, in both entertainment and industrial design fields. At launch, Sony will exclusively partner with Siemens to bring innovative immersive engineering capabilities to the manufacturing industry, through integration with the new NX™ Immersive Designer software, a product engineering solution from the Siemens Xcelerator portfolio of industry software. The integration of powerful hardware and software will empower creators in industrial design and product engineering field, by providing a more immersive design, review, and collaboration tools.

Sony will showcase a new product at our IGS during Metacenter Global Week in Orlando, Oct 17-19

Immerse Global Summit at Metacenter Global Week​ in Orlando on Oct 17-19!

In addition to it’s newest Spatial Reality Displays, Sony will also showcase mocopi at our IGS during ​Metacenter Global Week​ in Orlando on Oct 17-19!

Mocopi is a revolutionary phone-based motion 3D capture system for controlling virtual avatars that easily helps you track & record your full body motion; it’s great for your use in the metaverse! Mocopi is fully wireless and only requires a Bluetooth connection to your phone (iOS or Android) , so you can use it anywhere. It consists of six small sensors and a dedicated app that enable full-body motion tracking when combined using Sony’s proprietary technologies. Those in the industry know that traditional motion capture systems require pricey studios and trained operators, while Mocopi simply relies on Sony’s unique algorithm for accurate motion measurements with only small lightweight sensors as well as a smartphone.

Sony’s mocopi system makes motion capture and virtual content creation easy

With the dedicated “mocopi” app, users can create movies with their avatar in motion with their compatible smartphone, using the data obtained from the sensors attached to their body. In addition to pre-installed avatars, users can import custom avatars. Recorded avatar movies can be exported as mp4 files or motion data from the mobile app,” said Sony.

Mocopi is ultimately designed to record a user’s movements and then mirror them in digital environments — hence the mashup of “motion” and “copy.” There are plenty of different use cases for this kind of tech, from allowing animators to rig 3D characters with more realistic motions, to allowing Vtubers to replicate their movements in real time across streams and virtual reality platforms like VR Chat.

Mocopi provides some major benefits for the niche communities that will be willing to cough up the cash to buy it. While there are some affordable VR headsets like the Meta Quest 2 that can be similarly utilized in VR applications, these won’t provide the finesse of a dedicated motion capture tool, especially when it comes to lower body tracking.

Sony’s Mobile Motion Capture

Motion capture is a technique which digitizes the movements of a real person or object and imports them into a computer. This allows you to reproduce more lifelike, humanly movements with a computer-generated character in video production. This technique is also being widely used in movies, animations, and game contents within the Sony Group.

Typical motion capture requires studio facilities to install many cameras as well as a tight full-body suit worn by the actor with many markers attached to the body. In contrast, we have realized a new technology that enables motion capture using only small, lightweight sensors. We call this "Mobile Motion Capture."With this technology, you can readily digitize a person's movements while wearing everyday clothes whether they are indoors, outdoors, or anywhere and apply those movements to a computer-generated character.

If you haven't yet, get ​tickets​ to Metacenter Global Week.

Also, there is still time to become a ​sponsor​ or ​exhibitor​. Apply today to get the best speaking and or expo placement!

Sony investing $2B into XR (metaverse) research and development

Get to see the latest from Sony at our Immerse Global Summit at Metacenter Global Week in Orlando.

Sony, a member of the VR/AR Association (VRARA) dives into 'extended reality' with $2bn R&D war chest.

(Back in 2022, Sony already announced plans to expand into the Metaverse to boost its presence, by using its knowledge of gaming and music to build the Metaverse, allowing people to socialise, collaborate, and join gaming communities, among others. Also, Sony has invested roughly $1 B USD in real-time 3D software firm Epic Games Inc in a bid to develop its own Metaverse immersive worlds)

July 13, 2023 TOKYO -- Sony Group will ramp up R&D spending in its game business, as it seeks to hone technology in live service games and the emerging field of "extended reality" that blends physical and digital worlds. 

The company will pour 300 billion yen (about $2.13 billion) into research and development for its game segment for the fiscal year ending in March 2024. The amount, which will account for around 40% of its total R&D spending, now surpasses spending in electronics and semiconductors.

The figure is more than the 271.1 billion yen spent on game R&D last fiscal year, and more than double the 144.5 billion yen in fiscal 2020. Sony Group plans to spend 760 billion yen in company-wide R&D this fiscal year, up 3% on the year. 

Games are a major pillar driving Sony's business. Earnings before interest, taxes, depreciation and amortization -- or EBITDA -- for the company's game business was about 337 billion yen last fiscal year, up more than 60% from five years ago.

Sony's pivot from a focus on hardware to a subscription-based business model is spurring the company to step up R&D. Having long relied on sales of its PlayStation game console, the company will now focus on live service games that let customers purchase add-ons for titles streamed online.  

Sony projects that the market for add-on style games will reach $19 billion in 2026, a figure that would surpass the market for hardware like the PlayStation for the first time.

Yet Sony itself has yet to make a splash in live service games, relying mostly on sequels to traditional games for software sales. Meanwhile, Microsoft -- which already has a leg up in cloud-based streaming games -- is moving forward with a bid to acquire game publishing giant Activision Blizzard.

To chart a better growth strategy in the field, Sony in 2022 acquired major U.S. game studio Bungie for $3.7 billion, aiming to use its live game know-how to boost development.

Sony plans to have 12 live service game titles in its portfolio by the fiscal year ending March 2026, up from just one in fiscal 2021. 

It plans to allocate 55% of PlayStation 5 game development spending to live service games in the year ending March 2024, and 60% for the year ending March 2026.

The company also has its eyes on the metaverse.

Extended reality -- technologies that combine the physical and digital worlds -- is inching closer to the mainstream, with Apple's Vision Pro headset scheduled to hit the market next year. The technology promises immersive gaming experiences involving all five senses.

Sony aims to combine the resources of the nine overseas game studios it has acquired or invested in during the last two years to aid extended reality development. It has invested in U.S.-based Epic Games, whose "Unreal Engine" is one of the industry's leading computer graphics game engines.

"We will continue to make strategic investments going forward, prioritizing intellectual property," said Sony Group president Hiroki Totoki.

The company is also looking to bring the technology it cultivates in its game business into other fields, such as in-vehicle content for the electric car it is developing with Honda.

Sony's pivot comes amid a wider R&D push by other Japanese electronics companies, particularly in growth fields. The total R&D expenses of Japan's top eight electronics companies for the year ending March 2024 will increase 5% on the year to 2.28 trillion yen, reaching a record high for the third consecutive year.

Hitachi will increase R&D expenses in the environmental field by 27%, and Mitsubishi Electric will boost power-saving semiconductor development.

Source

Sony announced a new larger Spatial Reality Display

Sony announced a new larger Spatial Reality Display. A Larger 27 inch 4K Model, ELF-SR2, which provides a Realistic, Glasses-free 3D Visualization experience for professionals. For more info reach out to thaisa.yamamura@sony.com

PARAMUS, N.J., April 4, 2023 /PRNewswire/ -- Sony Electronics is adding a new model to its Spatial Reality Display portfolio with the addition of the ELF-SR2. At 27 inches, the larger 4K option provides highly realistic, three-dimensional content without the use of special glasses or VR headsets. Highlights of the new offering include an upgraded high-speed vision sensor, image quality enhancing technologies and installation flexibility. Additionally, the ELF-SR2 enables more robust functionality through its support of applications and development. It is optimized for industrial design, surgical planning, architecture, engineering, construction, signage, retail, software/application development, game developers and entertainment applications.

"Our original Spatial Reality Display continues to captivate customers by bringing 3D content to life with an astonishing authenticity and sophistication that demands to be experienced hands-on," said Rich Ventura, Vice President of Professional Display Solutions, Sony Electronics. "While our users love the technology, we keep hearing the same question from professionals; 'does this display come in a larger size?' As we expand into more B2B verticals and environments, I'm happy to say that in addition to a bigger screen, we've added some powerful and commonly requested features to enhance content production. These include a wider color gamut, a newly developed engine, more advanced high-speed sensors and rich support for applications and their development – all for an extremely competitive price."

At 27 inches, the 4K Spatial Reality Display provides highly realistic, 3D content without the use of special glasses.

Immersive Spatial Imagery
The display's visual fidelity is amplified by its immersive depth of field and detailed resolution, in addition to its larger size. New generation high-speed vision sensors enable high-speed processing, as well as low latency for reduced motion blur and crosstalk. 10-bit processing supports a wide color gamut that covers Adobe RGB at approximately 100% for accurate color reproduction. The super resolution engine provides upscaling from 2K to 4K. Additionally, the new model offers color moire correction to better address fine details, patterns and lines.

The ELF-SR2's enhanced facial tracking and recognition senses a viewer's eyes to provide a natural and comfortable visual experience, while the wide viewing angle enables consistency and accuracy from numerous vantage points.

Versatile Software Development Environment
The ELF-SR2 promotes application compatibility, as well as efficient and streamlined development through software development kits (SDKs). With support for leading SDKs including Unity and, Unreal Engine, it also allows for development with Open GL, DirectX11/12 and Open XR (coming later this year). The Spatial Reality Display enables simplified VR and AR digital content creation.

A new Spatial Reality Display App Select website will also be available, where users can easily find applications compatible with the display, as well as relevant case studies and information. In addition, Sony will offer an intuitive Spatial Reality Display Player app, which supports various 3D file formats to easily show 3D projects on the display. 

The Spatial Reality Display also allows for the use of specialized, industry-specific applications highlighted in the partnerships below.

Partnerships:

• 3DICOM MD, Singular Health's FDA-approved 3D software has been designed to work with Spatial Reality Display and enables highly accurate, detailed, glasses-free 3D medical visualization for diagnostic applications.

• Digital Nation Entertainment (DNE) provides end-to-end production and creative services for mixed reality and live experiences. DNE uses Spatial Reality Display to pre-visualize content from their volumetric capture studio.

• Developed by Arcturus, HoloSuite powers digital humans by editing, compressing and streaming volumetric video for virtual production and the metaverse, as well as AR/VR viewing on the Spatial Reality Display.

• KiksAR's platform brings real life shopping experiences to the digital world. Ultra realistic 3D configurators and personalized digital try-on experiences for jewelry and watch stores can be achieved in 3D using the Spatial Reality Display.

• Magnopus forges design, art, and technology to create new experiences across augmented reality, virtual reality and traditional reality. They're designing 3D entertainment content specifically for Spatial Reality Display and use it in their Virtual Production pipeline.

• Pixomondo (PXO) delivers Virtual Production and Visual Effects expertise and solutions for Film and Episodic content. PXO is using Spatial Reality Display to pre-visualize 3D assets built in Unreal Engine before final presentation on an LED volume, replacing a head-mounted display, for creators' comfort and convenience.

• SHoP Architects uses Spatial Reality Display for client presentations and has developed a mobile app to wirelessly control and navigate 3D models on it.

• Sketchfab is the leading platform for 3D and AR on the web. Using the Sketchfab API, Spatial Reality Display users will be able to download and view 3D models.

• WhiteMoon dreams, a leading independent game studio, uses Spatial Reality Display to evaluate 3D characters, creatures, textures, lighting, colors, and effects, instead of printing the 3D characters. This provides a faster and more sustainable real-time 3D evaluation.

User Friendly
The easy-to-use ELF-SR2 Spatial Reality Display's updated sensors and high-speed hardware processing provide accommodation for a wider variety of PCs to support different use cases and needs. With a detachable stand, the portable display can be installed in multiple different environments and configurations. It also accommodates the Video Electronics Standards Association (VESA) mounting standard for further flexibility and compatibility.

The ELF-SR2 is planned to be available in May 2023 in the United States and Canada through Sony's professional channels. It is expected to have a MSRP of $5,000 USD and feature a 3-year limited product warranty.

For more information on Spatial Reality Display, please visit https://pro.sony/ue_US/products/spatial-reality-displays. For new AV products and solutions from Sony Electronics, please visit https://pro.sony/prodisplaysolutions and https://pro.sony/press or follow Sony's professional business on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

Sony Electronics Inc. becomes a Member of our VR/AR Association

Sony Electronics Inc. is pleased to announce that they have become the newest member of the VR/AR Association. 

Both the VR/AR Association and Sony Electronics are dedicated to fostering growth in the virtual reality and augmented reality industries. As a member of the association, Sony Electronics will participate in a variety of initiatives by which they will be connected with VR/AR organizations around the world to accelerate the market with smart growth. In addition, Sony will be focusing on continuing to develop and build out technologies similar to their newly announced Spatial Reality Display. 

“We are excited Sony Electronics has joined our global association. Sony Electronics is an electronics leader in both the consumer and professional markets - constantly pushing the limits of innovation with their technologies,” says Kris Kolo, Global Executive Director, VR/AR Association. “We look forward to accelerating the market together with smart growth globally." 

As mentioned previously, the Spatial Reality Display from Sony is one of the company’s latest technologies to inspire this market. Using spatial reality to merge the virtual and physical worlds, the Spatial Reality Display delivers an impressive 3D optical experience viewable with the naked eye. This Spatial Reality Display’s 3D visual technology delivers a futuristic, yet highly practical visual experience where detailed colors, textures, contrasts, and brightness converge, forming a new creative medium for image, character and product design, visualization and manipulation.

“We are excited to participate in the VR/AR Association and to interact with fellow members as we explore ways to leverage our display leadership and heritage within the innovative VR/AR community” says Michiko Araki Kelley, VP, New Business and Corporate Marketing Group, Sony Electronics, North America.

For more information please visit www.thevrara.com.

###

About Sony Electronics Inc.

Sony Electronics is a subsidiary of Sony Corporation of America and an affiliate of Sony Corporation (Japan), one of the most comprehensive entertainment companies in the world, with a portfolio that encompasses electronics, music, motion pictures, mobile, gaming, robotics and financial services. Headquartered in San Diego, California, Sony Electronics is a leader in electronics for the consumer and professional markets. Operations include research and development, engineering, sales, marketing, distribution and customer service. Sony Electronics creates products that innovate and inspire generations, such as the award-winning Alpha Interchangeable Lens Cameras and revolutionary high-resolution audio products. Sony is also a leading manufacturer of end-to-end solutions from 4K professional broadcast and A/V equipment to industry leading 4K and 8K Ultra HD TVs. Visit http://www.sony.com/news for more information. 

About the VR/AR Association

The VR/AR Association (VRARA) is an international organization designed to foster collaboration between innovative companies and people in the virtual reality and augmented reality ecosystem that accelerates growth, fosters research and education, helps develop industry standards, connects member organizations and promotes the services of member companies.

Contact

Kris Kolo  kris@thevrara.com