For the third and final project of CS 491 AR/VR at UIC, my goal is to build a unified workspace that I am calling Collaborate. The idea of the project is that I have worked in many offices through internships. In each internship I have been at, I have seen one common problem: collaboration. When working with a new company, as an intern, you are bound to have questions and will need assistance regardless of how independent of a programmer you are. Most companies use Slack or some other chat service that is company wide. As that is great and all, once you get the attention of the person you are trying to work with, they come over and try and help. Then the person has to look through your code with you behind the laptop. The person trying to help does not have much control of what happens and s/he just tell's you to scroll up or down or look for certain things that could be the cause. This causes a lot of time to be lost on both sides; you lose time and the s/he loses time. Here is where Collaborate comes in to help.
Collaborate is going to be using augmented reality technology with the Hololens. The reason for that is that in an office space, even when you are wired in, you should still be able to see your surroundings and be aware of what is happening around you. With the Hololens, as a user, one can look around the room and still see all of his/her coworkers. The virtual reality experience would not be as good because your vision is blocked.
Collaborate would be used in a method that expediates the process of helping one another in an office while also respecting the time of both parties. Using Hololens provided by Microsoft, I plan to create an office space where everyone is in a Hololens (yes, I am aware that that would be one expensive endeavour) and they can interact with one another through the Hololens. So, as a company, I would have everyone in Hololens, with everyone having their own virtual desktop that they can work with. Each desk would have a Hololens, a mouse, and a keyboard. That reduces the need for having to carry around laptops or having desktops at each station, it reduces the need for a presenter at a meeting to bring their laptop or bring their presentation on a flashdrive or through email. This allows your virtual workstation to be a transportable area of work with ease.
Collaborate allows ever user to have their own workspace that they can choose how many monitors to have, what windows to show, what to hide, have some cool knicknacks on their desk. Collaborate is to help unclutter the workspace and make it more futureistic. As of now, the current idea is to only have a mouse and keybord on every desk because it is currently difficult to emulate a keyboard when it is not physical. Whether a mouse is needed or not, is up to the user; the user can use voice or touch controls or use a physical mouse. From that, a user can ask for help from a fellow colleague. Lets say that I ask for help from my coworker. From this point, my colleague can move or copy my window to his workspace. This would eliminate travel time as well as make it easier to collaborate with one another.
There will be one main user who can create new users. Each user will have their own workspace area. Each workstation is owned by a user, I will probably implement a basic login system that will load up a specific individual's workspace. Each individual workspace is locked to that user, but a user can share their workspace which will allow certain windows, in the workspace, to be able to be shared with fellow coworkers. Through that, it will allow people to collaborate from across the office! The main focus of this feature is to allow a futureistic collaboritive mode to allow coworkers to utilize each other in a more efficient way.
Collaborate will also allow for users to hide their workspaces to work in privacy. Sometimes, HR needs to work on compensation work and that needs to be done in private. Having a feature to enable users to work in privacy without having to worry about getting special screen reflectors or using special software.
Collaborate will also allow users to move their workstations into meetings and have a more open desk for people to work with. There will also be a virtual keyboard if a user wants to not worry about carrying around their physical keyboard. I am currently not sure about the features of a virtual mouse, but I would also allow for that to be used if a user wanted to use that.
The way that an office can manage their employees is that the office will all be a part of a virtual achor that is created by a reference point with Hololens. From there, as employees use their Hololens, they can find the anchor that was set up by their office admin and log in with their account to then see their virtual workspace. As of now, the current development of it will allow for a series of remote desktops to be run on the hololens. So, based on that, the only OS that is going to work natively would be Windows.
How this utilizes the AR space is that since each person is given their own unique workspace, a user can move their windows around, as menitioned before. But, you can also snap your workspace to another person's workspace to create a more collaborative space. This way, you can virtually see a coworker's workspace. This eliminates the reason for teams to sit together or for the office to be arranged a certain way. Making the office space and desk space cleaner by eliminating the need to have the office be arranged in a specific way. The physical area wouldn't be transfered but the augmented space would be able to be moved around to different areas of a given office space.
This allows for users to either push their work to other poeple's workspaces, or alternatively be able to pull their work into your space. This also allows for people to push or pull their workspaces to be able to virtually work next to someone to a better environment for peer coding or team collaboration.
For this project, I plan to use Microsoft's Hololens. The reason, as stated in the description, is that it allows for awareness of the environment. With that being said, I will be using Unity for my development tool because it is the most supported tool when comparing Unity and Unreal Engine. Unity has a lot of awesome libraries that can be used with Hololens made by Microsoft as well as various other people who seem to have extensive knowledge in augmented reality development, or at least understand augmented reality to an extend of being able to create tools for it. Some other hardware that I will be using is a physical keyboard and a physical mouse. Those two are because it currently seems difficult to emulate a keyboard in a nice, augmented way. Also, who doesn't like the clickity clack of a physical keyboard. The mouse is for a physical way of navigation rather than an augmented way of navigation within the virtual office. Aside from those, the main hardware I will be using is a physical computer to run the program to run Collaborate. In a perfect world, I would have some powerhouse computer that can house hundreds of Hololens, but since we don't have a super powerhouse computer or hundreds of Hololens headsets to test with, I will be using whatever UIC's EVL has.
The team consists of one member: Timothy Choh My role will be to develop Collaborate. I will be making the environment that users can work in as well as any UI that is needed to be developed.
The development process will be recorded here (Trello board)
I will start with creating a workspace that is easy to use and easy to navigate. From there, I will work on user experience and make sure that the application can be used by anybody that wants to use it. After that, I will be working on the centralized anchor to be able to share all the workspaces in the same space. *** MORE OF THIS WILL BE BROKEN DOWN AND SHOWN ON THE TRELLO BOARD
Microsoft Package I will be using: Github Link to MRTK
As noted about my project from the proposal, I wanted to create a work space that allowed for a virtual office where users can come to work and work at a desk but without the clutter of physical objects with the one caveat that the keyboards and mice were going to be physical because augmented keyboards and mice are from from being well made. From this project, I learned many things, including how to use the HoloLens and how to develop for the hardware. I learned that Microsoft is really pushing users to using Unity and Microsoft Visual Studio to develop their application for the HoloLens.
Some of the main things I had learned was that HoloLens have a sense of the world that it's in. It can read the physical dimensions of the space around it and build a virtual world around that. It was a unique experience to be able to try and fit virtual objects into a real world space. I also learned how companies use HoloLens to make their development of products faster and better. I learned that modeling in the augmented world is faster, easier, and cheaper than trying to do that all in the real world.
Those were some of the good things that I had learned from this experience, but there were also bad experiences with trying to develop for the HoloLens. One of which, is that I could not finish the project. I mentioned that Microsoft is pushing users to use Microsoft Visual Studios as well as Unity. I had many problems trying to get both environments to work, and once I did, problems kept persisting. I had multiple scenes for this project: one was for the login system where users would log in with a pin ( was trying to do it with username and password but did not have time to do that ). I also had a scene where the office managers would be able to setup different teams in their office, but that never fully came to fruition because the placement of open space within open space was much more difficult than I had originally thought it was. Within each team, there were users where they could setup their spaces to their left and right by selecting users within the office space. But the main problems came once I started trying to get all these things working together. Unity stopped showing the proper dropdowns that were needed, Microsoft Visual Studios wouldnt show my code because of outdated .Net runtime libraries even though Unity kept trying to make me downgrade them, and when I did update the .Net runtime, the dropdowns would not show the proper information either.
I had followed many tutorials, including those from Microsoft themselves, and it still wouldn't work. I had also followed YouTube videos to try and fix them. I reinstalled Microsoft Visual Studios many times as well as different versions. I had also install many different versions of Unity to see if it was a version issue, but I could not find the culprit in time of the due date for the project.
Overall, I am upset that my project did not work, but it's also a good learning experience. If I use hardware, it is bound to happen that something could be outdated and that something could go wrong with what I am working on. In terms of what the project was was that I had used Microsoft Academy as a base for my code to setup a sharable area where multiple users can see the same objects in the same space. Being able to share and hide objects based on a set of variables of objects. Another thing was that I realized that I could not get Remote Desktop to work properly with each user because I would have had to make my own client to run within a certain open space. That turned out to be much more difficult than I was originally expecting, but it was unique thing trying to debug. Another issue that came up was that I had to assign HIDs for a user for their keyboard and mouse. That would definitely not work very well in an office where there are hunderds of keyboards and mice all ove the office. The assets I currenty have are from the various Microsoft Academy tutorials that I went through as well as some default assets from MixedRealityToolkit. These were just placeholders because I was not sure how to implement a full desk with their tools in a nice way where the users could actually manipulate and play around with.
The uniqueness of trying to develop something for the HoloLens was not too far off except for the IDEs that were being used as well as limitations of hardware. If I could, I would definitely try and get a different type of experience working instead of having a collaborative office space. What I had originally thought of being a decent amount of work had very quickly turned into an excessive amount of work. In the end, I did not get a working product and did not have something to show off and play with.
The assets I had were from the two links found directly below this. I had built MixedRealityToolkit from scratch because of the Unity errors I was having, and yes I tried from source and from the download link that they had provided. I also used many of the assets from many of the Microsoft Academy tutorials. They were really good tutorials when they worked; they had good instructions as well as videos and all the code you need to follow along with the tutorials and take parts that I needed from each one to compile the project that I wanted to build. The main ones that I had used were 100, 101, 210, 211, 220, 230, 240, and 250. All of them yielded different parts of the world that I needed to create Collaborate.
MixedRealityToolkit I used: Github Link to MRTK ( Built from source )
Microsoft Academy link: Microsoft Academy link