MTEC3230 – Mixed Reality for Immersive Worlds

Syllabus

MTEC 3230, D296 – Mixed Reality for Immersive Worlds Thursdays, 2:15-5:35 pm, V103A & V314
Professor: Alexandre Girardeau
Email: agirardeau@citytech.cuny.edu
Office Hours: Thursdays, 1:30-2:00 pm
Office Location: V314
Office Phone (email is best): 718 260 5700

Course Description: This course will explore the new frontier of virtual, augmented and mixed reality across various professional uses and tools. Students will experiment with designing and developing game-based and interactive projects employing the latest AR, VR, and other reality capture-oriented devices. Students will learn the fundamentals of experiential design, Unity 3D development, VR art creation, and critical thinking in the context of storytelling and content creation specific to these emerging forms. Students will work in small teams on collaborative projects with the latest head-mounted and sensor technology.

Course Goals: Give students an introduction to: – the emerging field of virtual and augmented reality. – collaboration between people with different skills and backgrounds. – diverse design & development processes across various emerging platforms and media touchpoints.

Learning Outcomes: By the end of the course students will have gained: – Knowledge of key works in the emerging field of augmented and virtual reality art – Skills to use emerging AR and VR technologies not only as cool gadgets but as tools for meaningful aesthetic expressions, new narratives and critical commentary – A variety of practical proficiencies including basic knowledge of Unity3D as a software tool to create augmented and virtual reality art, and acquisition & translation of 3D scans of physical environments into immersive virtual reality environments that can be experienced and navigated with the Oculus Rift or HTC Vive headsets – Clearly communicate ideas using contemporary methods and critique work of their peers – Discuss the evolving technologies and innovative approaches to AR/VR used by professionals working across different fields, including the arts, social change as well as commercial applications.

Course Materials: – OpenLab – Flash drive & other portable drives or DropBox account to back up files – Readings (will be supplied for you as downloadable PDFs or links) – Sketch Book – Oculus account – GitHub account – Sketchfab account – Unity3D account

Expectations: – Spend at least 2 additional hours a week (outside of class) on class projects, readings & responses – Arrive on time and attend all classes – Follow good device etiquette – Silence phones & put them in your bag unless using for class assignment – Present in-progress and final work to your peers for critique – Participate in class discussions & group critiques – Back-up work every week

Lectures, Demos, Dos, Labs, Read & Respond: – Pay attention, take notes and ask questions during Lectures and Demos – React to Read & Respond as entries on OpenLab for dialogue (Before Class) – Do the dos to be prepared for the Lab time in class (Before class) – Ideate, Design, and Document your work via your OpenLab’s Portfolio (After Class)

Communication: – Contact your instructor with a brief, private question or message, via email (agirardeau@citytech.cuny.edu) – If you have a question that may be relevant to the group (about

homework, etc.), post your message via OpenLab – To discuss a longer matter with your instructor, send an email at agirardeau@citytech.cuny.edu to set up an appointment for office hours.

Attendance Policy: Attendance IS REQUIRED for all classes. If you have a legitimate reason for missing a class or assignment, or if you’ll be late, you MUST contact me before the class begins via email. All graded work is done collaboratively in a lab environment, and everyone’s objectives are interdependent. If you miss class, you will not be able to follow along or pass this class. ALL lab activities are graded by participation. Absence on these days will result in a 0 for the day. Project critiques are mandatory and cannot be made up. Missing a critique will result in a deduction of one letter grade for that module. See City Tech policy.

Academic Integrity Policy: Students and all others who work with information, ideas, texts, images, music, inventions, and other intellectual property owe their audience and sources accuracy and honesty in using, crediting, and citing sources. As a community of intellectual and professional workers, the College recognizes its responsibility for providing instruction in information literacy and academic integrity, offering models of good practice, and responding vigilantly and appropriately to infractions of academic integrity. Accordingly, academic dishonesty is prohibited in The City University of New York and at New York City College of Technology and is punishable by penalties, including failing grades, suspension, and expulsion. The complete text of the College policy on Academic Integrity may be found in the catalog.

Course Accommodations for Students with Disabilities: In order to receive disability-related academic accommodations students must first be registered with the Student Support Services Program (SSSP). Students who have a documented disability or suspect they may have a disability are invited to set up an appointment with Ms. Linda Buist, the program manager of SSSP (Phone: 718–260–5143, e-mail: lbuist@citytech.cuny.edu). If you have already registered with SSSP, please provide your professor with the course accommodation form and discuss your specific accommodation with him/her.

Grading: – 20% reading responses & portfolio – 10% VR/AR design document – 35% VR/AR development exercises – 25% final project – 10% participation & attendance

Grading Rubric:

Excellent (A: 90- 100) – Good (B: 80-89) – Fair (C: 70-79) – Unsatisfactory (F: 0-59)

 

Online Resources:
Unity Assets: VRTK, Oculus Integration, Oculus Avatar, NewtonVR, Unity Asset Store, Mapbox,
Template Projects: PlayoVR (multiplayer VR template), NormCore (multiplayer VR template)
Online info: XR Artists Collective, Facebook group VR Animation, XR Artists Toolkit,
Random: Google, YouTube, Looking Glass, Magic Leap, SteamVR, Oculus,
Apps: Unity3D, Blender, Ableton Live, Instant Meshes
Converters/Video Capture: OBS, Miro Converter,
3D Art: Sketchfab, Poly, Quill, Medium, Gravity Sketch, Tilt Brush, AnimVR, Tvori, VRSketch
3D Capture: Metashape, Mimesys, Rokoko
Communication: Airtable, Slack

News & Media Outlets:

 

Week Outline:

Week 1 – Thursday 1.30 – Introduction

Lecture: Overview of Course, Technology, Reality & Sensemaking
Lab: Setting up a VR headset (Oculus Rift)
Do (before class): N/A
Read & Respond (before class):

Endicott, S, 2020, What to Look Forward to in VR in 2020, Android Central, 01/01/2020. Retrieved from: https://www.androidcentral.com/what-look-forward-vr-2020

Fink, C, 2020, Mojo Vision Reveals XR Contact Lens, Forbes, 01/16/2020. Retrieved from: https://www.forbes.com/sites/charliefink/2020/01/16/mojo-vision-reveals-xr-contact-lens

 

Week 2 – Thursday 2.06 – Foundations of XR – Part 2

Lecture: Storytelling & Foundations of XR, Emerging XR Creative Tools, Selected XR Works
Lab: Setting up a VR headset and play area (Oculus Rift). Testing of installed VR apps
Do (before class): Collect 6 pictures (jpg, png formats only) that inspire you/your art. Copy them all in a folder on a usb stick or make it accessible online. Create an OpenLab portfolio, add your collected pictures, and briefly explain why you picked those pictures
Read & Respond (before class):

Knight, W, 2019, Enhanced Intelligence, VR Sex, and our Cyborg Future, Wired, 12/30/2019. Retrieved from: https://www.wired.com/story/enhanced-intelligence-vr-sex-our-cyborg-future/

Keslassy, E, Jan Kounen Discusses VR Experience ‘Ayahuasca’, Variety, 12/30/2019. Retrieved from: https://variety.com/2019/digital/global/jan-kounen-virtual-reality-ayahuasca-tribeca-film-festival-1203448679/

 

Week 3 – Thursday 2.13 –Semantics of Cyberspace & Human Nature

Lecture: Cyberspace, Human Perception & The Metaphysics of VR
Lab: Create a 360° interactive web-based environment using Wonda VR Spaces. Use your previously collected pictures as assets
Do (before class): Sign-up on Wonda VR Spaces (Use of Firefox is recommended).
Read & Respond (before class):

Rheingold, H, 1991, The Ontology of Cyberspace in Virtual Reality: Exploring the Brave New Technologies, Simon & Schuster, New York City (pdf here)

Heim, M, 1993, The Erotic Ontology of Cyberspace, in The Metaphysics of VR, Oxford University Press, New York. (pdf here)

Heim, M, 1993, Useful Vocabulary for the Metaphysics of Virtual Reality, in The Metaphysics of VR, Oxford University Press, New York. (pdf here)

 

Week 4 – Thursday 2.20 – XR, Ethics & Social Justice

Lecture: XR Ethics Manifesto, Uses of XR for Social Justice, XR in Public Spaces
Lab: Import your 3D assets in your Wonda VR Spaces project
Do (before class): Sign-up on Sketchfab, download +/-5 3D models (check the downloadable box in your filter while searching), unzip them and copy them in a folder on a usb stick or make them accessible online. Document chosen 3D models in your OpenLab’s Portfolio
Read & Respond (before class):

Noah Harari, Y, 2017, The Meaning of Life in a World Without Work, The Guardian, 05/08/2017. Retrieved from: https://www.theguardian.com/technology/2017/may/08/virtual-reality-religion-robots-sapiens-book

Burt, C, 2019, AI Now Calls for Ban on Affect Recognition as Market Expected to Surge to $90B by 2024, Biometric Update, 12/13/2019. Retrieved from: https://www.biometricupdate.com/201912/ai-now-calls-for-ban-on-affect-recognition-as-market-expected-to-surge-to-90b-by-2024

Yang, R, 2017, If You Walk in Someone’s Else Shoes, Radiator, 04/05/2017. Retrieved from: https://www.blog.radiator.debacle.us/2017/04/if-you-walk-in-someone-elses-shoes-then.html

 

Week 5 – Thursday 2.27 – XR Designing & Prototyping’s Essentials

Lecture: User Experience, Ideation & Development Process, Delivering
Lab: Ideation of a social justice-oriented XR experience
Do (before class): Find 5 articles on XR experiences related to cybersecurity, empathy and/or social justice, add them to your OpenLab’s Portfolio and briefly explain why you picked those
Read & Respond (before class):

Ben Lang, 2020, Varjo’s Workspace Demo is a Glimpse of VR’s Long-Term Future in Workplace, Road To VR, 01/10/2020. Retrieved from: https://www.roadtovr.com/ces-2020-varjo-workspace-demo-vr-workplace-future/

Emrich, T, 2019, Augmented Reality Experiences Will Save Brick & Mortar Retail, Medium. Retrieved from: https://medium.com/near-future-of-retail/augmented-reality-experiences-will-save-brick-and-mortar-retail-3ae4e946d67c

 

Week 6 – Thursday 3.05 – Building for VR with Unity3D [DEMO]

Demo: Create a VR experience in Unity3D for Oculus HMD
Lab: Create your VR experience in Unity3D for Oculus HMD Part 1
Do (before class): Document the ideation process from the previous lab class in your OpenLab’s Portfolio
Read & Respond (before class):

Assor, J, Pohl, R, 2019, Level Up Your VR Skills, Unity Blog, 22/13/2019. Retrieved from: https://blogs.unity3d.com/2019/11/13/level-up-your-vr-skills-with-a-free-new-course-from-unity-and-oculus/

 

Week 7 – Thursday 3.12 – XR Sharing Platforms & Formats Essentials

Lecture: Review of XR Sharing & Resourceful Platforms, Study of Different Tools & Formats
Lab: Continue your VR experience in Unity3D for Oculus HMD using the plugin Mapbox
Do (before class): Document your Unity3D workflow on your OpenLab’s Portfolio
Read & Respond (before class): N/A

 

Week 8 – Thursday 3.19 – The Rise (and Fall?) of VR, AR, 360° Videos & 3D Audio

Lecture: Current State of XR, Introduction to 360° Video Capture and Development
Lab: Creation of a volumetric video capture using a Kinect and the plugin Mimesys
Do (before class): Document your predictions on the rise and fall of XR technologies (+/- 200 words)

Read & Respond (before class):

Hardaware, D, 2019, 2020 is VR’s Make-or-Break Year, Engadget, 12/24/2019. Retrieved from: https://www.engadget.com/2019/12/24/vr-in-2020-oculus-quest-half-life-alyx/

Farsace, B, 2019, Insta 360 1 X Review: I Love This Camera But I’ll Never Use it, The Verge, 07/30/2019. Retrieved from: https://www.theverge.com/2019/7/30/20732263/insta360-one-x-review-360-camera-video-price-specs-features

Week 9 – Thursday 3.26 – Building for AR with Unity3D [DEMO]

Demo: Porting for AR with Unity 3D (plane, image and face detection) using AR Foundations
Lab: Creation of an AR experience using Unity 3D (Port for iOS and/or Android)
Do (before class): Document the workflow of the previous lab class in your OpenLab’s Portfolio

Read & Respond (before class):

Wilson, T, 2017, The Principles of Good User Experience Design in Augmented Reality, Medium, 11/13/2017. Retrieved from: https://uxdesign.cc/the-principles-of-good-user-experience-design-for-augmented-reality-d8e22777aabd

Lukic, D, Seol, P. 2016, Virtual Art, Anchored in Reality: A Conversation on Location-Based AR. New Media Caucus, 11/01/2016. Retrieved from: http://median.newmediacaucus.org/mestizo-technology-art-design-and-%20technoscience-in-latin-america/virtual-art-anchored-in-reality-a-conversation-on-location-based-ar/

Week 10 – Thursday 4.02 – Mapping & 3D Capture

Lecture: State of 3D Capturing technologies, Marketplaces & Best Uses
Demo: Creation of 3D scans with the Structure Sensor and Skanect, Photogrammetry with Agisoft Metashape
Lab: Create a 3D Scan using your phone’s camera and Metashape
Do (before class): Find 3 articles related to the class’s subject (3D mapping & capture) and document in your OpenLab’s Portfolio.

Week 11 – Thursday 4.09 – Draw, Sculpt & Animate in VR [DEMO]

Demo: Demo of Painting/Animating in VR with Tilt Brush & Quill, VR Sculpting with Medium
Lab: Recreation of your chosen object/environment in Tilt Brush or Medium
Do (before class): Pick an object or environment/artwork you will reproduce in VR using painting tools. Briefly explain your choice in you OpenLab’s Portfolio
Read & Respond (before class):

Girardeau, A, 2018, A Conversation with Matt Schaefer: VR as a Design Tool, Animation Nights, 04/10/2018. Retrieved from: https://www.animationnights.com/vr-design-tool-matt-schaefer/

Week 12 – Thursday 4.23 – Mixed Reality Video Capture [DEMO]

Demo: Mixed Reality Video Capture using a Kinect and LIV
Lab: Creation of a Mixed Reality Video Capture using a Kinect and LIV
Do (before class): Document your workflow from the previous lab class in your OpenLab’s Portfolio. Brainstorm ideas for your final XR project

Week 13 – Thursday 4.30 – Final Project

Lab: Ideation and prototyping, Work on Design Document
Do (before class): Document your workflow from last class in your OpenLab’s Portfolio

Week 14 – Thursday 5.07 – Final Project

Lab: Gathering Assets & Level Design, Finalize Design Document
Do (before class): Document your workflow from last class in your OpenLab’s Portfolio

Week 15 – Thursday 5.14 – Final Project

Lab: Physics & Scripting (if needed), or testing and further development work
Do (before class): Document your workflow from last class in your OpenLab’s Portfolio

Week 16 – Thursday 5.21 – Final Project

Lab: Porting to XR Platform/Device
Do (before class): Document your workflow from last class in your OpenLab’s Portfolio

Week 17 – Thursday 5.28 – Final Project

Lab: Testing, Sharing, and Critiques
Do (before class): Document your workflow from last class in your OpenLab’s Portfolio