Note – you can find a pdf of the course syllabus HERE, but be sure to check back at this page for any updates!

MTEC 3230, OL45– Mixed Reality for Immersive Worlds

Thursdays, 2:15-5:35 pm,

Professor: Sky Rolnick • Email: srolnick@citytech.cuny.edu Office Hours: Thursdays, 5:35-6:35PM EST

Course Description: This course will explore the new frontier of virtual, augmented and mixed reality across different market sectors. Students will experiment with designing and developing game-based and interactive projects employing AR, VR, wearables, IoT, and machine learning for mobile, web and console environments. Students will learn the fundamentals of Unity development, 3D modeling, stereoscopic perception and experiential design in the context of storytelling and content creation specific to these emerging forms. Students will work in small teams on collaborative projects with the latest head-mounted and sensor technology.

Course Goals: To give students an introduction to: – the emerging field of virtual and augmented reality. – collaboration between people with different skills and backgrounds. – diverse design & development processes across various technology platforms and media touchpoints.

Learning Outcomes: By the end of the course students will be able to:

  • Knowledge of key works in the field of immersive media arts
  • Understand implications of VR/AR for the humanities, policy and social sciences
  • Basic knowledge of Unity 3D to create VR/AR experiences
  • Ability to clearly communicate ideas and think critically
  • Problem solving using online tools
  • Understand key works in the emerging field of augmented and virtual reality art.
  • Use emerging AR and VR technologies not only as cool gadgets, but as tools for meaningful aesthetic expressions, new narratives and critical commentary
  • Employ a variety of tools, including basic Unity3D to create augmented and virtual reality art.
  • Acquire & translate 3D scans of physical environments into immersive virtual reality environments that can be experienced and navigated with the Oculus Rift or HTC Vive headsets
  • Clearly communicate ideas using contemporary methods, and critique work of their peers
  • Discuss the evolving technologies and innovative approaches to AR/VR used by professionals working across the fields of commercial, artistic & social change.

Course Materials:

  • OpenLab
  • Slack: mtec3230fall2021.slack.com (https://join.slack.com/t/mtec-3230-fall2021/shared_invite/zt-vkwfhdy8-5Gm25~9pN0R04M5cfdKrFg)
  • Readings (links available on Openlab and in the syllabus)
  • Flash drive & external hard drives or Dropbox to backup your files
  • Openlab: openlab.citytech.cuny.edu/groups/mtec3230-mixed-reality-for-immersive-worlds/
  • Readings (will be supplied for you as downloadable PDFs or links)
  • Sketch Book
  • Steam account
  • GitHub account

Expectations:

  • Spend at least 2 additional hours a week (outside of class) on class projects & readings
  • Upload weekly HW by Thursday at 8 pm or by date due listed on OpenLab
  • Arrive on time and attend all classes
  • Follow good device etiquette. Silence phones & put them in your bag unless using for class assignment
  • Present in-progress and final work to your peers for critique
  • Participate in class discussions & group critiques
  • Back-up work every week
  • Check Slack regularly for group and private messages
  • Reach out to students and your instructor with questions
  • Push creatively and technically. Stay open and curious

Lectures, Demos & Labs:

THINK – Prepare reading responses & journal entries on blog for discussion (before class)

DO – Finalize Design & Development Exercises & Digital Prototypes by Documenting on Portfolio (After Class)

Attendance Policy: Attendance IS REQUIRED for all classes. If you have a legitimate reason for missing a class or assignment, or if you’ll be late, you MUST contact me before the class begins via email. All graded work is done collaboratively in a lab environment, and everyone’s objectives are interdependent. If you miss class, you will not be able to follow along or pass this class. ALL lab activities are graded for participation. Absence on these days will result in a 0 for the day. Project critiques are mandatory and cannot be made up. Missing a critique will result in a deduction of one letter grade for that module. See City Tech policy*:

Grading:

20% reading responses & VR/AR design journal

20% VR/AR development exercises

15% AR Project

10% VR design document

25% final project

10% participation & attendance

Grading Rubric:

VALUES Excellent (90-100) Good (80-89) Fair (70-79) Poor (60-69) Unsatisfactory (0-59)
Concept Core concept is intriguing, original, and well-explored Core concept is intriguing but lacking in examination Core concept is present and supported by the work Core ideas are scattered without consideration No clear concept, or work doesn’t reflect it
Progress Clear and consistent progress from ideation to execution Progress was made, but was not consistent Evidence of procrastination, “last minute” pushes or crunch Lack of progress in 1-2 areas resulting in project deficiencies Little to no progress shown on the project
Presentation Concept is clearly presented and strongly supported through audio, visuals, interaction, and narrative (if applicable) Concept is supported through presentation, but 2 or more areas of the design are lacking or distracting Concept is weakly supported through presentation, project requirements met at a “bare minimum” level 1-2 presentation requirements are not met. 3+ presentation requirements are not met.
Skills Clear demonstration of skills in all development areas (visual, text, audio, interaction, programming) Clear demonstration of skill in 2+ development areas Demonstrates skills, but omits topics covered in class. Evidence of skills, but underutilization of techniques learned in class Does not use any techniques learned in class.
Collaboration Consistently provides honest, supportive feedback to peers, responsible in meeting team goals, communicates effectively. Generally supportive, responsible, and good communication, with a few issues Multiple issues/problems with collaboration, meeting goals, or communicating Little to no evidence of communication, goal setting, and collaboration in a team setting.   Disrespectful to fellow students work, with negative impacts to class/team dynamics.

Academic Integrity Policy: Students and all others who work with information, ideas, texts, images, music, inventions, and other intellectual property owe their audience and sources accuracy and honesty in using, crediting, and citing sources. As a community of intellectual and professional workers, the College recognizes its responsibility for providing instruction in information literacy and academic integrity, offering models of good practice, and responding vigilantly and appropriately to infractions of academic integrity. Accordingly, academic dishonesty is prohibited in The City University of New York and at New York City College of Technology and is punishable by penalties, including failing grades, suspension, and expulsion. The complete text of the College policy on Academic Integrity may be found in the catalog.

Week Outline:

Week 1 [FOUNDATIONS]- Thursday 08/26 – Introduction

Lecture: Presentations, course overview, communication tools & equipment, syllabus.
Lab: Introduction to Creative VR Apps (Mozilla Hubs, Medium, Quill, Gravity Sketch).

Read & Respond: https://www.theguardian.com/technology/2017/may/08/virtual-reality-religion-robots-sapiens-book (respond to prompt question on the blog)

Week 2 [FOUNDATIONS]- Thursday 09/02 – Immersive Art 101 + Source Control

Lecture: What is immersive art? Is immersive art the same thing as interactive art? Immersive art in a covid-19 era?
Lab: Pushing a new Unity project to Git from Fork. Unity 3D basics Part 1 (interface, camera, lights, skybox, intro to C#, rotate script.

Do (before class):
– In your Openlab portfolio, briefly document an immersive experience that you really enjoy/has impacted you (+/-10 lines).
– Create GitHub & Unity3D accounts if you don’t already have one.

Week 3 [FOUNDATIONS]- Thursday 09/09 – Critical Perspectives on Technology + Unity Basics Part 1

Lecture: Unity Events, Observer Pattern, delegates, Actions, functs, and review of ON_Tools a collection of scripts. Review Terrians, ProBuilder, PostProcessing, and Road Architect.

Do (before class):
– Download and install Unity (Follow the same procedure I showed you last week) if not already done. Create a scene in Unity and change the camera’s skybox for a solid color. Add a plane, a cube, a sphere and assign different materials to them. Take screenshots of your scene and post on your portfolio.
– To go further (optional): do you know how to use a custom skybox? Import and set up your own skybox.
– Feed your portfolio with pictures from 4 movies/animes that inspire you/your work. Briefly explain what you like about them (10 lines for each max).


Week 4 [FOUNDATIONS]- Thursday 09/23 – XR Media Archaeology + Unity Basics Part 2 + Guidelines for Project 1: Portal

Lecture: Media archeology of the latest XR devices and their relation to storytelling.
Lab: Unity 3D – We will continue to review c# scripting, print & debug, character controller, animation controllers, and unity particle systems. We will develop project from last week and create a portal system. Guidelines for Project 1: Portal.

Do (before class): 

Create a scene in Unity 3D, create a terrain with grass, mountains, trees & grass, import a custom skybox. Add a structure you’ve built using Probuilder. Add a cube that changes colors when you click on it. Add post effects to the camera and post screenshots of your scene on your portfolio. Briefly explain the look/atmosphere you’ve tried to achieve (10 lines max).
–

Week 5 [FOUNDATIONS]- Thursday 09/30 – Unity Basics Part 3 – C# Scripting 

Lecture: C# Scripting & online resources (input method, if, loop, instantiate, random), Scripting exercises, Unity Video Recorder & animating the camera.

Read: https://immerse.news/making-a-new-reality-625b78aeb1c5

Experience: Find a user-created experience in VR that you really enjoy, write a journal entry or blog

Do (before class):
–  Add a character controller to your scene. Create a custom particles systems (special effects, rain, snow, fog, galaxy, etc.). Take screenshots of your scene and post on your portfolio.

Week 6 [FOUNDATIONS]- Thursday 10/07 – C# Scripting – Deliver Project 1: Portal + AR Foundations + Unity XR Interactive Toolkit

Read (optional) https://arvrjourney.com/the-principles-of-good-user-experience-design-for-augmented-reality-d8e22777aabd 

Lab:  Unity Project templates for AR Foundation and XR Interactive Toolkit


Do (before class):

Read through your some of your classmate’s pitch decks here.  As one of the following roles (DO NOT play as yourself!), what game interests you the most and why? come up with a few questions to ask the person pitching the game.

ADDITIONALLY, for those of you with VR setups, you can begin testing VR apps and taking notes on what you like and don’t like about various VR experiences.

Week 7 [DEV SKILLS]- Thursday 10/14 – Creative process & Storyboarding +  Coding for Web XR and Unity XR

Lecture: Creative process, design and storyboarding, Lab: Storyboarding exercises. Web XR, 8th Wall, A- Frame + Glitch, Artivive, Mozilla Hubs(and https://sketchfab.com/ for assets). Generating Markers & Mapping to World.

Do (before class):
Publish to github a portfolio project to demonstrate AR Foundation and/or XR Interactive Toolkit.

Read & Respond (before class): 

Babich, N, The Role of Storyboarding in UX Design, 10/25/2017, (link here)

Week 8 [DEV SKILLS]- Thursday 10/21 – Guidelines for Project 2: VR/AR (Group Project)

Lecture: Use of XR technologies in entertainment, industry and education. Developing 3D environments, adding sounds and ambience. Designing Spaces and Audio in VR Space

Guidelines for Project
Lab: Lab time for Project 2 (team git-ready, research & sketches).

Do (before class):
– Share a link or video of a Web XR experience.

Week 9 [DEV SKILLS]- Thursday 10/28 – Current State of AR + AR Foundations

Lecture: Current state of AR, Review of Project 2’ GDDs.
Lab: Project development with Q&A covering Unity tools and Web XR. .

Do (before class):
– With your group, ideate a game-based immersive experience and prepare a first version of your GDD. Document the process and the evolution of the project (from ideas to proof of concept) on your portfolio. All team members must update their own portfolio.

Week 10 [DEV SKILLS]- Thursday 11/04 – Current State of VR + Oculus SDK

Lecture: Current state of VR, UI in VR.
Lab:  Project 2: Proof of Concept.

Do (before class): 
– Work on your proof of concept scene for Project 2 (core mechanics, level design). Document and post screenshots/short videos showing progress on your portfolio.

Week 11 [DEV SKILLS]- Thursday 11/11 – Presentations of Proof of Concept for Project 2: VR/AR

Lecture: Presentations of your proof of concept for Project 2, peer critics.
Lab: Lab time for Project 2 based on feedback from presentations.

Do (before class):
-Prepare a 10min slide show presentation of your proof of concept for Project 2 (including a 2min live demo). Make sure your GitHub page is up to date with a readme file and GDD. Share screenshots of the current status of your project on your portfolio.

Week 12 [BUILD & DEPLOY]- Thursday 11/18 – Live Video App & Audio Visualizer

Lecture/Lab: Audio Visualizer in Unity & live video app using Agora SDK.
Lab: Lab time for Project 2: Alpha version.

Do (before class):
– Work on your Project 2 alpha version (core mechanics, level design). Document and post screenshots/short videos showing your progress on your portfolio.

Week 13 [BUILD & DEPLOY]- Thursday 12/02 – Support for Project 2: VR/AR

Lecture/Lab: Photogrammetry/3D scans, 3D imports/exports with Blender/Unity.
Lab: Support & Lab time for Project 2: Alpha version.

Do (before class):
-Further development of your Project 2 alpha version (UI, scripting, refine assets). Document and post screenshots/short videos showing your progress on your portfolio.

Week 14 [BUILD & DEPLOY]- Thursday 12/09 – Final Presentations

Do (before class):
– Finalize the alpha version of your group project with all core features working.
– Create a short video trailer (30 seconds minimum – 1 minute max) with voice over or sound effects. It should look like a trailer for a video game, make it interesting.
-Share your build (File, Build Settings/Add open scenes, build) as a zip file with the class on the slack channel.
-Prepare a +/- 15 minutes slide show presentation of your group project (this includes the video trailer, as well as a 2 minutes live demo).
– Mention who did what in the project (it should be mentioned in your project’s timeline/GDD anyways). Even if you failed implementing a feature, mention it.
– The slide show presentation should be a summary of your GDD, illustrated with reference images and screenshots, etc.
-Share screenshots and video trailers of your alpha version on your portfolio and Slack.

Week 15 [BUILD & DEPLOY]- Thursday 12/16 – Feedback Session

Lecture: Review of main concepts studied, feedback session.
Lab: Extra lab/presentation time if needed.

PROJECT 1: PORTAL:

-Create a portal in a Unity environment. Experiment with world building, and create a stylized concept. Document your creative process and game pitch on your portfolio. Upload/Push your project on Git. Record a walk through of your scene using the Unity Video Recorder, briefly edit the video, and share it on your portfolio and Slack.

PROJECT 2: AR/VR: CREATE A MEDITATIVE EXPERIENCE

-In groups of 2, create a meditative AR or VR experience that responds to user’s inputs. Work collaboratively remotely using Git & Fork. Document your creative process, GDD, and full workflow on your portfolio. Create a 30sec/1min video trailer with voice over, sound effects, title page, etc. and share it on your portfolio and Slack. 

Resources:

  • Road to VR: http://www.roadtovr.com/
  • VR Focus: http://vrfocus.com/
  • Articles by Re/code’s Eric Johnson: https://www.recode.net/authors/eric-johnson
  • The New York Times Business and Technology sections, and Bits blog: http://bits.blogs.nytimes.com/(you can sign up for free with school email)
  • The VR Geeks list: http://www.vrgeeks.org/vr-geeks/global
  • Engadget: http://engadget.com
  • Twitter hashtags #vr, #ar #oculusrift, #htcvive. People constantly share cool stuff there and via other hashtags.

Unity 3D Plugins/Assets/Packages [ALL FREE]

Note: This list will be constantly updated, please check again and feel free to contact me to suggest plugins or tools you’re interested in working with.

Other resources

News & Media Outlets