There haven’t been any longform blog entries in quite a while, though you may find newer short entries on my twitter: https://twitter.com/getnamo
For the past couple of months I’ve been working on bringing TensorFlow to UE4, enabling the use of state of the art machine learning in Unreal Engine projects. The plugin has reached something I think is ready to try out, but it is very much a work in progress so please check it out and give some feedback if you can!
There’s the plugin
and the examples project
There are pretty detailed instructions at the repositories outlining how to use the plugin with examples.
What examples you ask?
Basic Add/Subtract float arrays
A simple example that shows how to pass data into python, running tensorflow with the data and then passing results back out to UE4
Mnist Recognition using UE4 textures
Maybe you want to recognize hand written digits using UE4 textures? How about drawing shapes in your browser?
Upcoming Pong example
Not quite ready to use yet, but soon there will be a pong AI example, trained using basic q learning
Which one will be the champion?
Try it out and pop any feedback you may have on https://forums.unrealengine.com/community/work-in-progress/1357673-tensorflow
I had the awesome opportunity to join a team with Matt Burri and Mark Bailey for the VRFocus Virtual Reality Challenge. A 24 hour session of frantic coding, meowing, and last minute hacks to get something playable.
Our team cobbled together Catastrophe, a subversive multiplayer game where you have to save falling kittens while waiting to betray your fellow coop partner. Here’s an early singleplayer gameplay video:
The game only really picks up its true form in multiplayer though, which is what we presented at the end of the countdown as the winning entry. VRFocus covered their event extensively, their articles, a recommended reading.
My favorite part of jams is always the end when you get to try what the other teams did; inspired entries, each with their own unique take on the theme, in this case ‘A safe pair of hands’.
Apart from being crazy fun, there were a ton of challenges which we later covered as a presentation during the latest London UE4 meetup.
Check out the slides for more information.
We’re hoping to take this further, keep up to date with the project at our website catastrophevr.com
For the third London is Unreal meetup I made a presentation on the topic of plugins and chose to implement a useful plugin live during the presentation. A socket.io client was chosen as the plugin, which was finished with a basic Connect, Bind, Emit functions and an On event all interfaced through an Actor Component. Doing this in a small time frame was a daunting task, but something I was certain was doable. While I didn’t meet my target of 15min for the live implementation, within 25minutes the whole meetup was chatting away from their phones with a UMG chat widget in the game engine.
The presentation slides can be found here.
How does this look like in blueprints?
After adding the new plugin component to your actor of choice
You connect and bind functions of choice
which you can receive as events
and emit equally easily
Made a quick experimental port for using the Razer Hydra in VR
This branch of the hydra plugin won’t be maintained, but allows you to try out the VR editor if you don’t have a Touch or Vive lighthouse motion controller
See the experimental branch for details on how to get it running: https://github.com/getnamo/hydra-ue4/tree/vr-editor
Since 4.11 the community plugin I’ve maintained has been included in the engine. Simply enable the plugin in your project to begin using it
Then change your game mode to use the convenience character to see your hands like this
For quick setup: https://developer.leapmotion.com/unreal#quick-setup
Latest documentation: https://github.com/getnamo/leap-ue4
Further work on the NexusVR application has required me to find a solution to handle zip files. Having found no suitable alternatives, I decided to port 7zip to UE4. This resulted in the ZipUtility plugin.
Fully multi-threaded and exposed to C++ and Blueprint, it allows for easy extraction and progress update.
From 7zip support for many file formats are available such as Zip, 7-Zip, GZip, BZip2, RAR(decompress only), TAR, ISO, CAB, LZMA, and LZMA86. This and some utility windows file API functions means that it should serve my needs well and I hope you UE4 developers will find good use for it!
The concept was a DK2 portaling app that allowed you to transit between VR experienced you loaded in your folder. The original goal was a multiplayer social portaling hub, but due to time constraints the multiplayer aspect of it got dropped. What did make into the final version though was a fully enabled chromium browsing experience which you could browse using your hands in VR.
If you wanted to share something you were browsing to the big screen a simple reach into the browsing window (depth interaction) would convert the current page into a data cube. This data cube could be easily transferred to any visible surfaces which would render the link or video.
Mac Cauley’s visuals created a vibrant landing area that will be the core meet up place in the future social version!
If you have a DK2 and a Leap motion, check it out at: nexusvr.io
I’ve recently implemented a fully rigged collision blueprint in my unofficial leap plugin. The way it is constructed allows for you to change both the collision asset and mesh to suit your needs, while retaining the Leap Motion based input.
I’ve also added easy IR passthrough support in the Leap Motion plugin, allowing you to transition from Virtual Reality to Augmented Reality with a simple gesture. Now you can reach for that beer without removing your headset!
The architecture is finally starting to separate the input from the actual output and it should be relatively easy to have other input drive the rigged mesh.
which is the next step, but for now let’s try out the collision accuracy on some Jenga
I may need some practice, but it is great to see that finger collision is accurate enough to do delicate separation. Leap motion tracking has come a long way!
The Myo plugin has also been featured on Thalmic Labs’ Myo Unleashed series. Its a very concise entry on how to add Myo support to your game in UE4 in under a minute. Let me know if it helps!
See the link below for the blog post: