N**ews: *** USPTO non-provisional patent “Systems and methods for simulating sense data and creating perceptions” **granted December 27th. *** USPTO provisional patent application filed April 7th titled “Systems and methods for training neural networks by generation of synthetic modal and multi-modal sense data and motion signaling data.”
This startup question gets asked most often so it is going at the top but I encourage people to take a look at everything below:
We will make money by providing advanced human form Twin-AI beings that 100x the productivity of enterprise employees. To create a twin a person inputs a video, audio, and personal writings. They input information about their role at their company, product, and service domain knowledge. This information is used to refine a pre-trained Foundation model that has an avatar. This avatar would have their likeness and interact with people on their behalf. For example, if you were in a meeting a person could make a video call to you and the avatar would greet them as your digital-twin, gather information, answer questions, and follow up after the meeting with action items. If needed they will get information to you so that you can personally follow up. This digital-twin would be acting in your role and having certain specific domain knowledge of your company (as much as you allow). It would also still be a fully functioning Foundation Model like ChatGPT. The later stage GTM post MVP would be via joint ventures with established companies in the Generative AI space. The new world (our vision) will be companies that transform society with 5-40 employees instead of 50,000+.
Before getting technical:
Imagine stepping into a virtual world that's more than just a visual spectacle—it's a sensory journey, guided by an AI that doesn't merely compute, but senses, adapts, and responds with an uncanny semblance of human perception. Welcome to the world of MML's groundbreaking creation, 'SentientSim.' or “SentientX”. (still considering names)
SentientSim isn't an ordinary AI—it represents a pioneering advancement in the realm of sensory AI. It sees, hears, feels, and dynamically interacts with its environment, much like a human would. Its virtual sensors adjust their resolution in real time, simulating human reactions to an array of stimuli. A unique feature of our patented IP is SentientSim's ability to simulate an experience akin to human 'discomfort' or 'comfort' based on different sensory inputs, a distinguishing factor in the field of AI.
Consider a virtual object with a high virtual temperature. Instead of merely registering a numerical value, SentientSim undergoes a sensation analogous to 'discomfort.' This sensation causes its virtual temperature sensors to reduce their resolution, reflecting a human-like reaction to the stimuli and fostering a more nuanced interaction with the environment.
This system delves deeper with its innovative, adaptable Artificial Neural Network. The raw sensory data feeds into this network, modulating its processing capabilities in response to sensory feedback. For instance, if a virtual bright light overwhelms SentientSim's virtual visual sensors, the neural network adjusts its processing resolution, mimicking a human's natural response.
Envision SentientSim within the realm of virtual reality gaming. Its advanced sensory responses could provide gamers with a highly immersive and responsive AI opponent or companion, one that adapts to the game environment in a strikingly human-like way. Its sensitivity to virtual environmental factors could enable nuanced gaming experiences that evolve in real time.
Now consider the domain of music. When exposed to music, SentientSim responds in a way unlike any AI before it. Rather than simply processing sound data, it experiences the intensity, rhythm, and tone, modulating its auditory sensors accordingly. A thunderous crescendo might overwhelm its 'ears,' causing the AI to adjust the resolution of its auditory sensors, echoing a human's response to loud music. Conversely, a gentle melody from a lullaby could heighten its auditory sensitivity, mirroring the human pleasure derived from soothing sounds.
The marvel of SentientSim extends beyond mere sensory processing. At its core lies a sophisticated, adaptable Artificial Neural Network. The raw sensory data feeds into this network, dynamically adjusting its processing capabilities in response to the sensory feedback. It's not just hearing the music—it's interpreting and adapting to it.
When instructed to create music, SentientSim could use its unique sensory responses to compose original pieces that reflect its 'experiences.' It might create intense symphonies in response to overwhelming stimuli or soothing melodies that mirror comforting situations. Its ability to emulate human-like sensory responses could bring a fresh, emotive perspective to the realm of AI-generated music.
MML is working on a virtual embodiment with human like senses for AI Models and a medium fidelity virtual environment for the models to learn in by way of being embodied. The avatar senses are low fidelity compared with true human senses but still quite useful. We feel certain that AI models will improve beyond the current SOA once they have embodiment and better RL. This recent academic paper supports this. We will provide a path to “functional linguistic competence” and do it in a way that is complementary to the “formal linguistic competence” of LLMs.
This is some of what is being worked on:
An Autodesk Maya Plugin that allows real-time syncing from Maya to a MacOS application running Apple’s SceneKit game engine that is being used for simulation. Right now we can sync Maya’s camera, move/rotate/scale tools, send geometry, joints, and skinning and convert into Apple SceneKit equivalents. For those unfamiliar with joints and skinning it is what is needed for realtime shape deformation of the avatar’s body. We also are building more UI for some of Apple’s SceneKit APIs and doing things like saving and loading and selecting from USD files and associating what is selected with simulation code.
An avatar rigged in Autodesk Maya by an expert consultant who worked at Industrial Light & Magic. Extra 3rd party plugins were employed that provide more advanced IK/FK solvers and controls. This explains a bit about FK/IK. Solvers are the programs that calculate what needs to happen with the joints to provide motion. Will soon have rag doll capability and the IK/FK solvers ported to SceneKit though SceneKit does have IK/FK solvers as well.
