Micheal, Rahul, Daniel and I came together to form the Reality Rebels team at the Meta + AWS Hackathon in Seattle this week. Tear your room’s walls apart with a portal & shoot targets reminiscent of Zelda BotW! We used threlte (threejs + Svelte) to create the experience. A big shoutout to Micheal Parks for his great work integrating and maintaing the XR package for threlte, which was the base for our project! He also came up with a cool soundtrack for our game, all within the hackathon timeline! While the original vision of the game included multiplayer aspects using WebRTC (which we partially got working), I had lots of fun regardless! I also got to meet with lots of talented people and learn about their experience working with WebXR and MR tech!
You can also find my post on Twitter / X.
]]>As the WASI API gets a more solid shape, WebAssembly (shortened to WASM) has immense potential to become the ultimate cross-platform build target. Taking the concept forward are languages like AssemblyScript that make compiling code in the browser look easy!
Want to see the power of WebAssembly? See this epic project by Peter Salomonsen which offers a flexible way to create music right in the browser, including realtime MIDI input support! The project page is here.
While the editor on the AssemblyScript homepage does include a way to share the code you are working on, it would be nice to share it in a more legible format, say through GitHub Gists. This would however mean OAuth authentication in a simple client / frontend setup, so no storing of client secrets! To make it worse, GitHub doesn’t allow CORS authentication.
GitLab to the rescue! GitLab supports the increasingly recommended and secure OAuth2 PKCE flow, including CORS authentication! I have a simple integration in place already here that allows an authenticated user to create public and private snippets. As I am not storing the access and refresh tokens yet, the creation flow is a bit unintuitive, requiring the user to press Create Snippet through the share menu, login through GitLab and then press the Create Snippet button again, this time with the correct content in the files. Fixing this, persisting file and project details between refreshes (through localStorage) and snippet loading are on my radar. You can find the full repo on GitHub.
]]>I have always been a big fan of the different 3D UI explorations by UltraLeap (previously Leap Motion). One of them involving Project Northstar, the open HW/SW passthrough AR headset from Leap Motion is very special to me.
I want to create something similar, but only this time it will all be in WebXR! I am also experimenting with adding a watch that can stand in for almost any rectangular smart watch, as I am scaling a square model based on the required aspect ratio. To make this even more dynamic, I also have an attachment point in the watch’s model where I can insert any watch face!
Maybe in the future I will be able to make the watch face completely dynamic instead of a static image - this and many other wonderful ideas can come to life when WebXR DOM Layers spec is ready! For now I will settle with creating watch faces for my very real Amazfit band 7.
]]>Ever since I read about Hermione’s fake galleons that could be used to communicate the next meeting’s date and time in Harry Potter and the Order of Phoenix, the idea has made a permanent house in my mind. My previous attempt to create an AR “fake-galleon” app based on a QR-code physical token/shared secret led to me learning quite a few things about SubtleCrypto and making a REST API using the minimalist Python backend framework, Falcon, but I never completed it. While I might pick it up another day, I wanted to see if I could make something with my new Amazfit Band 7 fitness band/ smart watch. Borrowing ideas from old-school pager buzzes and Facebook’s feature to “poke” a friend as a way of getting their attention, I set out to create a set of apps that would enable this. I wanted to tap a button on my watch, which would send a message to the Zepp Android App, which should then be able to somehow send a notification to all connected devices?!? As I had already created some simple apps for the watch like Random Cat Fact fetcher and other watch faces, I knew I could reuse that code for the watch app side. So what remained? Well, almost everything major.
I needed a way to send updates to all the registered devices. A PWA that could show notifications should do the trick. I looked into how I could send notifications to connected devices, which introduced me to the world of WebPush, Firebase Cloud Messaging and VAPID authentication schemes. I wanted to create my own API endpoints that devices could register to and then receive VAPID-secured notifications.
The server and client codelabs from web.dev came in really handy at this point, as it had the base Express server with some default endpoints set up to register and push notifications to connected clients. I wrapped up the client into a PWA, complete with custom icons, and set up some other endpoints to send customized messages within the PWA (while still allowing for a default notification from devices like a smart watch). I asked a few people from my family and friends to test it out and it seems to work! Find the full source to the app on the watch, server and PWA on my GitHub.
]]>I have been working on a creating simple but functional 3D UI in three.js for the AR passthrough mode. The passthrough mode became available for WebXR apps only recently, following which I added it as a feature to Mozilla’s Jumpy Ball and my music synth tool, DAW[n]XR. But I wanted to create a 3D UI demo that could be re-used in other projects as a base. Hand-tracking and AR passthrough work quite well together so I chose to focus on integrating these two features. As I went through the examples in the three.js repo, one of them having functional buttons and hand tracking posed itself as the perfect base for this project.
The code for this example is a bit on the complex side as it uses ecsy (which is also used by Mozilla’s Jumpy Ball and the recent WebXR demo from Meta called Project Flowerbed. My first step was to separate out the javascript portions to separate files. I also created separate module files for systems
and components
. Finally I started working on a hand-attachment script that would help me attach arbitrary Object3D to various joints on the arm.
This is when I encountered a (small) bug involving initialization of hand-joint objects. While I expected this to happen as soon as the hand data was available in a session, it turns out the joints were created only in the next update frame! Going around this would have meant checking the availability of this data in an update loop till it became available, but why do that when you can propose changes yourself? As three.js is an open-source library, I dug into the code and created an MR addressing the issue. Till the fix is accepted and released, I will stick to a locally built version of the library that incorporates this change. I will be writing another article soon with updates. Stay tuned!
]]>Link to the game hosted on Github
Inspired by the latest Meta Connect 2022 Talks on WebXR, I searched for WebXR games using Entity-Component-System (ECS) patterns. And there it was, Jumpy Balls, a fun WebXR physics platformer game from Mozilla Mixed Reality Team. While I remembered seeing it a couple of times in the Oculus Browser showcase, now I became genuinely interested in it! A fun game with all code and assets out in the open? I am sold!
Setting up the development environment was a bit of a pain as the dependencies in the latest commits did not play well together. My initial thoughts were to make the relevant upgrades to support the newer packages but that seemed to take forever. I even explored moving from webpack
to vite
in hopes of simplifying the bundling steps. Just as I was getting edgy, I found out the root cause of the confusion: the older commits were using a deprecated version of ecsy-three which had a weird version number. I used the frozen version numbers from package-lock.json
as I ran npm ci --legacy-peer-deps
and the repo built fine! This was a great learning experience for me in terms of solving dependency issues, a simple guide with the original repo would have saved me quite some headache though… Anyway, now you can place platforms around inside your room as you try to get the ball to cross the target hoops!
The next step was to incorporate the WebXR passthrough changes that I fiddling with over the last two days. This was surpisingly easy once I got the dev environment set up, possible made easier by the ECS structure of the game.
I hope to add some new features to the game in the coming weeks. I was excited to see that the game already comes with a hidden level editor (that you can access by adding ?edit
to the URL), but there is no way yet to make any persistent levels. This just the kind of game for creating funky levels in a level editor! It would be great to share the levels using QR codes ala Mekorama:
Wait, I know, I know, I will leave you with the link to the GitHub repo before you start pointing the virtual cannon at me! I have opened a few issues in the repo hoping to get contributors interested. See you there!
]]>Meta released this after their recent Meta Connect 2022, during which they showcased a WebXR experience running in full color passthrough on the new Meta Quest Pro. I tinkered with the three.js paint sample to enable passthrough. The main changes are:
This is my first time using threejs to make a WebXR app, other times I either used PlayCanvas or BabylonJS. I also used a different environment to host the example this time around - CodeSandbox. Full code here
]]>New shader experiements with the BabylonJS node material editor, again playing with the idea of waves - Subtle Glow and Glowing Band. The effects work well across different objects in the shader preview dropdown like the sphere, the plane and the cube. As usual, the colors are variables that can be fully customized. This time around I created a custom node with code inside that acts as a smooth bandpass filter:
{
"name": "Smooth Bandpass",
"comments": "Allows a band of values to pass through, smoothly clamping the rest to zero",
"target": "Neutral",
"inParameters": [
{
"name": "inValue",
"type": "Float"
},
{
"name": "bandWidth",
"type": "Float"
},
{
"name": "center",
"type": "Float"
},
{
"name": "smoothWidth",
"type": "Float"
}
],
"outParameters": [
{
"name": "output",
"type": "Float"
}
],
"functionName": "smoothBandpass",
"code": [
"void smoothBandpass(float inValue, float bandWidth, float center, float smoothWidth, out float result) {",
" float leftEdge = center - (bandWidth * 0.5);",
" float rightEdge = center + (bandWidth * 0.5);",
" float leftStep = smoothstep(leftEdge - smoothWidth, leftEdge + smoothWidth, inValue);",
" float rightStep = 1.0 - smoothstep(rightEdge - smoothWidth, rightEdge + smoothWidth, inValue);",
" result = min(leftStep, rightStep);",
"}"
]
}
More about custom nodes in BabylonJS on the docs page.
]]>It had been a long time since I had a chance to redesign my website, so I dove straight in. While the design is pretty simple, I created a few sketches using Indesign Studio, followed by a lot of cutomization of the Bulma Clean Theme Gem for Jekyll. I used the base colors and layout of the Darkly theme for Bulma too! I hope this has made my website more polished looking and readable!
]]>
The Node Material Editor from Babylon JS team is a intuitive Node based Shader editor that works in the browser! It also has nifty organizational features to create custom frames and nodes! I created two shaders to get a feel of the workflow - one uses Simplex Noise to create a lively blob, while the other uses clever vertex displacement to generate waves along the surface. Try them in the browser, make sure to select the Sphere in the shape dropdown on the right to see the intended effect!
]]>