What NYU Future Reality Lab does

NYU Future Reality Lab is an academic research laboratory housed within the NYU Courant Institute of Mathematical Sciences, founded in 2017 and directed by Ken Perlin. The lab focuses on mixed reality (XR/VR/AR) technologies for human-to-human communication and collaboration, producing open research artifacts and peer-reviewed publications rather than commercial products. Its project portfolio spans Chalktalk (an MIT-licensed cross-device digital presentation and drawing tool that underpins several downstream projects), Metaroom (a multi-user mixed-reality room-scale environment), Holojam (a shared holographic social XR system), GazeChat (gaze-aware communication), Robotecture (robot teleoperation), VRGaitAnalytics (vision-based clinical gait and vestibular analysis), DrawTalking (a multimodal world-building system co-developed with Adobe Research and published at UIST 2024), and a Conditional Diffusion-based generative terrain authoring pipeline that won the ACM SIGGRAPH VRST 2024 Best Paper Award.

The lab operates 11–50 personnel including faculty, postdocs, and graduate students at the NYU campus in New York City, and has produced 30+ publications at top-tier venues including SIGGRAPH, UIST, CHI, and IEEE VR. Technically, its stack combines Unity-based XR applications, real-time motion and gaze tracking via computer vision, neural rendering for avatar synthesis, and generative AI (Conditional GANs and Conditional Diffusion) for procedural environment authoring. Its business model is institutional: the lab is funded through NYU, federal research grants, and selected industry research partnerships (notably Adobe Research), and distributes software under permissive open-source licenses rather than selling licenses or subscriptions. Output is consumed by the academic research community, clinical collaborators in vestibular and balance rehabilitation, and open-source developer/users of Chalktalk.

Name

NYU Future Reality Lab

Legal name

Future Reality Lab

Website

https://frl.nyu.edu

Company type

Private

Founded year

2017

Operating status

Operating

Headcount range

11–50 employees

Short description

NYU Future Reality Lab is an academic research lab at NYU Courant, directed by Ken Perlin, that builds open-source mixed-reality (XR/VR/AR) tools for human communication and collaboration, serving the research, education, and clinical communities through grants and partnerships.

Ownership category

akta.pro rank

Product category

Mixed Reality Research Laboratory

NAICS

Research and Development in the Physical, Engineering, and Life Sciences (54171), Research and Development in the Social Sciences and Humanities (54172)

SIC

Services-Commercial Physical & Biological Research (8731), Services-Computer Integrated Systems Design (7373)

akta.pro primary industry

Immersive Collaboration Classrooms (Multiuser VR, Social Learning Spaces) (EDAFAJAK)

akta.pro secondary industries

Creator Tools & Content Pipelines for Immersive Learning (VR lesson builders, 3D asset workflows) (EDAFAJAL), Digital Twins & Scenario-Based Systems Training (Equipment/Process Simulators) (EDAFAJAJ)

Keywords

• Mixed reality research
• Virtual reality prototyping
• Augmented reality interfaces
• Human-computer interaction
• XR collaboration tools

Headquarters

HQ city

New York

HQ country

United States

HQ region

North America

Offices1 record

Markets served

GTM type

B2B

Offering type

Services

Cost components

Personnel, Technology or R&D, Operations

Revenue model

1. Research Funding: Academic research lab funded through NYU and research grants. No commercial revenue generation.

Go-to-market motion1 record

Distribution channels1 record

Marketing channels6 records

Core offering

NYU Future Reality Lab is an academic research laboratory that develops mixed reality (XR/VR/AR) prototypes, tools, and research outputs focused on improving human communication and collaboration in shared physical-digital spaces. Core offerings include the open-source Chalktalk presentation system, VR-based gait/balance assessment tools, and research prototypes on shape-changing interfaces, gaze-aware communication, and multimodal world-building, released as open-source software and academic publications rather than commercial products.

Product overview

NYU Future Reality Lab is an academic research laboratory focused on developing prototypes and tools for mixed reality communication and interaction. The core research outputs include Chalktalk (an open-source digital presentation system), VRGaitAnalytics (a VR-based gait assessment platform), and various XR research prototypes including DrawTalking for multimodal world building, collaborative physical design environments, shape-changing interfaces (Robotecture), and tangible/graphical interfaces for large object manipulation. The lab also explores applications in healthcare including vestibular rehabilitation and balance assessment. Most projects are released as open-source or academic publications rather than commercial products.

Differentiator

Problem solved

Functional benefit

Products and services

• Chalktalk A digital presentation and communication language that uses a blackboard-like interface to create and interact with animated digital sketches in 2D and 3D for live presentations or conversations; released under MIT Open Source license on GitHub.
• VRGaitAnalytics A low-cost VR gait assessment system that simulates virtual obstacles and visual/auditory/cognitive loads while using motion tracking to assess walking performance, with in-situ spatial visualization for trial playback and real-time outcome measures.
• DrawTalking A multimodal research prototype for building and controlling interactive worlds by sketching and speaking while telling stories, enabling programming-like capability without requiring code.
• Collaborative Multimodal XR Physical Design Environment An XR system with multimodal input capability providing annotations and interactive visual modifications by superimposing and aligning visual counterparts to physical objects, enabling rapid prototyping and design iteration without physical modifications.
• Robotecture A cost-efficient and expandable shape-changing system using modular robotics with actuated support beams that generate dynamic surface displays and enclosures through linear movements, enabling tangible interactions across different scales.
• Tangible and Graphical XR Interfaces for Large Object Manipulation A comparative study of tangible user interface (TUI) using 3D-printed Opti-Tracked miniatures and 2D graphical user interface (GUI) minimap for manipulating large virtual objects in XR environments.

Quantifiable outcome

• 12 out of 15 vestibular patients showed significant improvements on Visual Vertigo Analogue Scale (p=0.02), Dizziness Handicap Inventory (p=0.008), and Activities-Specific Balance Confidence Scale (p=0.02) after VR training
• +1 more outcomes

Named customers2 records

Segments3 records

Ideal customer profiles3 records

Technology focussed Yes

API detail

Has API

No

API docs

API detail

Core technology

AI maturity

App detail

AI capability9 records

Feature8 records

Partnerships

Three partnerships are on record, tiered core and minor.

• Adobe ResearchcoreStrategic or Co-development Partner · 1 January 2024Collaboration on DrawTalking research project; Adobe Research co-authors include Rubaiat Habib Kazi, Li-Yi Wei, and Haijun Xia; published at UIST '24
• HolojamminorStrategic or Co-development PartnerCollaborated on 'Holojam in Wonderland' - the first multi-audience, multi-performer live action theatrical performance completely in VR, presented at Future of StoryTelling (FoST) festival October 3-8, 2017
• NYU Courant Institute of Mathematical SciencescoreStrategic or Co-development PartnerHome institution providing academic infrastructure and affiliation; lab is part of NYU CIMS as indicated by @NYU Courant link

Scale indicators3 records

Recent moves5 records

Expansion highlights4 records

Emerging players

• Magic Leap: Magic Leap is a commercial mixed-reality platform pursuing enterprise and creator use cases (spatial computing, Magic Leap 2) with collocated collaboration features that compete for the same users and design problems FRL targets in its Metaroom prototypes.
• Niantic Labs: Niantic operates an AR platform (Pokémon GO, 8th Wall) with a research arm studying shared real-world AR experiences, comparable in problem framing to FRL's collocated mixed-reality communication work, though it ships at consumer scale.

Others

• Unity Technologies: Unity is the dominant XR development platform whose engine and authoring tools underpin much of the XR prototyping space (including FRL-style research projects), making it an adjacent enabling/commercial layer comparable to Chalktalk's creator-tool positioning.

Broad incumbents

• Meta Reality Labs: Meta Reality Labs is the dominant commercial incumbent operating in the same collocated and social XR space (Quest hardware, Presence Platform, codec avatars), with vastly more capital and shipping product while FRL operates as a research counterpart producing open prototypes in the same problem space.
• Microsoft Research (Mixed Reality): Microsoft Research's mixed reality group publishes at the same academic venues (CHI, UIST, SIGGRAPH) and ships HoloLens/Mesh, overlapping with FRL's interest in collocated multiuser XR, asymmetric collaboration, and creator tooling but at much larger scale.
• Google AR/VR Research: Google's AR/VR research arm works on creator tooling, spatial presence, and education applications (Google Expeditions/Classroom) overlapping directly with FRL's Metaroom, Chalktalk, and asymmetric-VR-education work, but at commercial scale.

Direct peers

• Adobe Research: Adobe Research is an explicit FRL collaborator on DrawTalking (UIST '24) and operates a similarly structured industrial research lab publishing at top HCI venues focused on creative tools and authoring, making it both a partner and a comparable research organization.
• MIT Media Lab: MIT Media Lab is the most direct academic peer: a similarly positioned university research laboratory that publishes at top HCI/XR venues, develops open-source prototypes (e.g., fabrication and wearable tools), and cultivates industry partnerships in mixed reality and human communication, mirroring FRL's role at NYU Courant.
• Stanford HCI Lab: Stanford HCI Lab is a like-sized academic group publishing at the same top venues (CHI, UIST, SIGGRAPH) on VR/AR interaction techniques, asymmetric collaboration, and creator tooling, directly comparable to FRL's research output and venue footprint.
• Carnegie Mellon Future Interfaces Group: CMU's Future Interfaces Group is a small academic research group focused on novel XR/AR interaction techniques, sensing, and haptics with strong industry partnerships and publications at the same venues (CHI, UIST), making it a close functional peer in scope and scale.

Market position

Strengths5 records

Weaknesses5 records

Competitive moat3 records

Key risks5 records

Key highlights7 records

Customer concentration

• Xx.com/nyufrl
• Facebookfacebook.com/nyufrl

Revenue estimate

Valuation estimate

Number of profiles

Profiles5 records

Funding overview

Funding rounds

Investors

Funding detail is available on the Subscription and Enterprise plan.Contact sales →

M&A

Investments

M&A and investment is available on the Subscription and Enterprise plan.Contact sales →

Frequently asked questions about NYU Future Reality Lab