By virtually re-creating and visualizing an intelligent space (iSpace) geographically far from the original version, one opens up the possibility of r... Robots - Three dimensional displays - Mobile robots - Robot kinematics - Glass - Visualization - Data visualization - control engineering computing - delays - middleware - mobile robots - telerobotics - 3D visualization - intelligent space - time-delay compensation - remote controlled environment - iSpace - virtual copy - stereoscopic visualization tool - computer network time delay - mobile robot - 3D stereoscopic visualization - Intelligent Space
3D Visualization for Intelligent Space: TimeDelay Compensation in a Remote Controlled Environment Peter Zanaty*, Drazen Brscic†, and Zsolt Frei‡ *Department of Automation and Applied Informatics, Budapest University of Technology and Economics, Budapest, Hungary †Institute of Industrial Science, University of Tokyo, Tokyo, Japan ‡Institute of Physics, Eötvös University, Budapest, Hungary [email protected], [email protected], [email protected]
Abstract — By virtually re-creating and visualizing an intelligent space (iSpace) geographically far from the original version, one opens up the possibility of remote control and interaction between the real iSpace and the virtual copy. Here we present the required components for such a mixed environment: the actual iSpace in a form of a laboratory built for such experiments, the stereoscopic (real 3D) visualization tool at the far end to reproduce the original iSpace, and the middleware embodied by the computer network that connects the two. To remotely operate in the iSpace one has to overcome the important problem of synchronization that is made difficult by the time delay in the computer network. Here we present our first approach to solve this problem by demonstrating a successful remote control of a mobile robot in this mixed environment. Keywords — 3D stereoscopic visualization, Intelligent Space
recent years we saw an active development of the emerging research field of spaces that have distributed sensors for monitoring the space. These spaces have various names, such as smart spaces or ambient intelligence, ubiquitous computing, whereas in our research we have been using the name Intelligent Space (hereon iSpace). iSpace is a room (corridor, street, or any other well defined environment), which has distributed sensory intelligence and is equipped with actuators. For a review, see . There are a number of research approaches similar to iSpace. Easy Living Technologies  utilizes human tracking technology with the purpose of guessing the intent of users in the space in order to automate and facilitate everyday tasks. The Oxygen Project  is a similar research aiming at the development of intelligent environments based on human-centered computation. The Interactive Workspace Project  is yet another attempt in which the possibilities of using ubiquitous embedded sensors and information displays are explored. Not all use robots as physical agents. A small number of research approaches (for example , , and ) use them the way we do in our iSpace. N
Finally, there is a considerable effort of research on sensor networks, ubiquitous computing , and mobile robot control, which is also more or less related to the iSpace initiative. The main objective of an iSpace is to provide services to humans inside the iSpace. Often mobile robots are introduced into an iSpace as actuators. Tracking humans and tracking and controlling robots in iSpace is a complex task, for a review of current approaches see . In this paper we introduce yet another possible extension on the concept that is being developed: remote control and interaction with an iSpace. Sensory data may need to be transmitted to a remote mainframe computer for processing and control data may need to be sent back to agents in an iSpace. Moreover, remote control by humans of a distant iSpace may be necessary and optimal. Central control rooms could help out those in temporary need in iSpaces, without the requirement of the constant local presence of controllers. Such a remote control introduces the problem of time-delay. We connected an iSpace (located in Japan) to a remote location (in Hungary) where a stereoscopic projection system aided the remote interaction with the iSpace in Japan and provided real 3D feedback of a position of a mobile robot in the remote iSpace. We demonstrated by controlling the mobile robot remotely that the problem of time-delay could be solved with a simple approach. In § 2 we describe the iSpace (in Japan), in § 3 we outline the stereoscopic projection system (in Hungary), in § 4 we present the required tools for connecting the two above (for remote control and visualization), and in § 5 we detail our simple predictor that was successful in overcoming time-delay problems. We offer our conclusions in § 6. II. ACTUAL ISPACE - IN JAPAN To verify the concept of iSpace, an experimental room was established in Japan . The experimental room is shown in Fig. 1. The iSpace consists of eight CCD
cameras, an ultrasound positioning system, and two mobile robots (P2-DX) manufactured by ActiveMedia Robotics.
Fig. 1. Experimental iSpace at the University of Tokyo, Japan. Many sensors gather data and two mobile robots (red colored on the left hand side of this picture) act as actuators in iSpace. In our experiments we did not use the CCD cameras, but collected and transmitted data from the ultrasound positioning system. This ultrasound positioning system consists of small sized transmitters located on the mobile robots and about 90 receivers mounted on the ceiling and around the room. To achieve high positioning and orientation accuracy, and to ensure fast system response, the ultrasound positioning system is coupled to the mobile robot’s wheel encoders. We used Extended Kalman Filters (hereafter EKF) , both to filter noisy ultrasound data, and to filter wheel encoder data that might be affected by wheel slippage. To carry out our experiments, this laboratory was connected though the internet to the remote visualization system described below.
our case here). Commercial applications, such as the recent development of a 3D movie camera  and the expected emergence of 3D enabled TV sets in the near future capable of displaying movies created with this new device  will bring the technology to the mass market in a very short time. The fundamental idea behind all stereoscopic solutions is that to create a real 3D sense of space for the human observer, one has to provide left-eye-perspective and righteye-perspective images for the viewer’s left and right eyes, separately. The creation of these images from slightly different perspectives is a simple task of rendering for today’s computers, since the description of the 3D space is assumed given. The only challenge is the physical separation of the two images. One image must only be seen by one eye, and the other image is intended solely for the other eye. Current solutions all utilize one of the physical properties of light to achieve this separation. Anaglyph glasses use the spectral property of light to separate the images: red and blue color filters are used in front of the left and right eyes so that red-colored and blue-colored images are separated. Unfortunately, this system will not reproduce color images with high quality. The proposed solution to this problem are the filters introduced by Infitec: these are special filters with three transmission windows in the visible part of the spectrum, transmitting light in parts of the red, green and blue regimes. Color reproduction is possible with these filters and left-eye- and right-eye-image separation is possible since the transmission windows do not overlap in the left-eye and right-eye filters .
III. VISUALIZATION CENTER - IN HUNGARY The Visualization Center (hereafter VC) at the Faculty of Science, Eötvös University, Budapest, Hungary was founded in 2006. One of the major projects undertaken was a construction of a large stereoscopic projection system with the primary aim of providing a platform for scientific visualization. This stereoscopic system is well suited for the application that is the subject of the current paper. It provides and immersive and collaborative environment for visualization, therefore it is very useful for visualizing an iSpace at a distance. There are a plethora of solutions for stereoscopic (real three dimensional, hereafter 3D) visualization at the time of writing of this paper. Computer generated data often provide 3D information that created the need for the development of these systems. Examples include results of scientific simulations (such as hydrodynamics, or the large scale structure of the Universe), a 3D description of an object (i.e. a large molecule consisting of many atoms) or the detailed 3D description of an intelligent room (as in
Fig. 2. Four possible technical solutions for stereoscopic image-separation: Anaglyph glasses use color filters to separate images (a). Infitec glasses use advanced filters that can reproduce color images by wavelength multiplexing (b). Passive stereo uses polarizing filters with different orientations (c). Active stereo uses LCD shutters to separate images in time (d). Passive stereo utilizes the fact that light can be polarized. Linear polarizing filters (vertically and horizontally oriented in front of the two eyes in a glass frame) separate the left-eye and right-eye images, that are both projected on the same screen with the appropriate polarization (which is usually achieved buy using two
projectors for the two images with corresponding polarizing filters in front of the projector lenses, or using a single projector and a polarizing filter the can change the direction of polarization). Active stereo separates images in time. Left-eye and right-eye images are projected in alternating sequence (one for the left eye, one for the right eye, and so on) at a very high frequency (preferable at 120 Hz or above, so that there are at least 60 images per second for each eye). An active stereo shutter glass (using LCD shutters) synchronized to the projector ensures that only the targeted eye sees the image projected at any given time. We summarized these solutions – picturing the glasses they use – in Fig. 2. Systems for scientific visualization almost all use active stereo today. Arguably they provide the best image quality, although at an expense: active LCD shutter glasses are very expensive compared to passive polarizing glasses. In case of scientific visualization environments, where the number of simultaneous viewers - and thus glasses needed at the same time - is limited (usually less than a dozen) the cost of LCD shutter glasses is not prohibitive. However, in movie theaters, where the number of spectators is large, passive stereo is used more often.
In the VC at Eötvös University our stereoscopic system consist of only one wall. This is a cost effective solution, requiring less projectors, but still providing a projected image that fills the viewing angle of the observer almost entirely. This system may not qualify as a CAVE, but besides the number of walls, the other characteristics are there: it is built in an underground, flat-black painted dark room and the large screen is back-projected by highresolution projectors. There are actually two projectors, that both illuminate the same wall. They provide a very wide aspect ratio screen (2.36:1) through edge blending. Each projector has a native resolution of 1400 by 1050 pixels. The two images, projected side-by-side, overlap in the center by 320 pixels. Both projectors send information to these overlapping pixels, but projector power is adjusted in this blended region so that total pixel illumination is always constant. The smooth transition between the two projectors renders the overlap virtually invisible, and provides an 2480 by 1050 pixel single image on a 5m by 2.1m screen. Since this resolution is high, the projection is from the back, and the screen-size is life-like, the user can move close to the image, where the wide screen fills even his peripheral vision. He is almost totally immersed in the virtual environment, thus the system being cost effective in function comes close to what a CAVE does. We do not have motion tracking in place yet. The system provides images from a fixed viewpoint at the moment. A tracking system based on ultrasonic technology is in the final stages of development, and will be installed to aid our future experiments.
Fig. 3. A typical CAVE. There are three back-projected walls (front, left and right) and one that is projected from above (the floor is “down-projected”) in this system. Wired tracking and input devices complete the setup. There are many Cave Automatic Virtual Environments (hereafter CAVEs) in operation today. A CAVE is an immersive virtual reality environment where projectors are directed to three, four, five or six of the walls of a roomsized cube, usually from the outside. They provide total immersion for the user. The CAVE definition, first introduced in 1992  specifies • high-resolution projectors, • back-projection systems, and a • dark environment to minimize possible reflections. • The user is usually tracked, and computer generated images based on the position of the user in the cave provide the feeling that objects float in space and can be walked around.
Fig. 4. The stereoscopic projection system at the VC of Eötvös University. Active stereo is used to create real 3D images. The screen is large, back-projected and uses a very wide aspect ratio so that the user feels immersed in a virtual environment. Ultrasonic tracking (with receivers hung from the ceiling and emitters mounted on glasses) is under development. (The room is lit here so that system components can be seen. Normal operations take place in an unlit room.)
IV. ADDITIONAL SYSTEM COMPONENTS Humans receive most of their information visually; significant part of the neurons in the human brain is dedicated to visual processing. 3D visualization of information is probably the best way of feedback to the operator. This can be achieved using active stereoscopic rendering with the system described above, but generating one image per eye in order to enhance the depth perception in the virtual world is needed by the computer system. This type of rendering is supported by special graphics hardware and software, which can draw into four buffers instead of the usual two, and can display eye specific images in sync with the LCD shutter glasses. In order to utilize the capabilities of the special hardware, we modified and adapted a well-known open source graphical engine for image rendering, called OGRE 3D . The capabilities of the engine we selected are compared to two other leading open source engines in Table 1.
The application developed for providing feedback for the operator uses the OpenGL rendering subsystem and runs in a Linux environment. The virtual equivalent of the iSpace in Tokyo has been created. A 2D rendering of the virtual laboratory can be seen in Fig. 5. We used the virtual equivalent of the real iSpace in Hungary to project a 3D virtual version of the iSpace on the large screen at the VC. We conducted remote control experiments between Hungary and Japan during March, 2008. The VC stereoscopic screen with the virtual iSpace projected is shown in Fig. 6.
TABLE 1: COMPARISON OF IMAGE RENDERERS Criteria OGRE 3D CrystalSpace OpenGL Approach
Fully object-ori. Extensive
Object-oriented programming, readability, changeability and maintainability of the source code - which is important in creating a solid framework - were taken into account, when choosing OGRE as our rendering platform. OGRE has also already proven to be industrial-, education-, and commercial-ready in hundreds of successful projects. There are several projects that use OGRE for 3D visualization, but there is no existing native support for stereo rendering in the framework. In order to enable the stereo rendering mode, the OpenGL rendering subsystem of OGRE has been altered and Quad-Buffer support was introduced .
Fig. 5. Here a 2D rendering of the 3D model of our iSpace (in Japan) is shown. One mobile robot is pictured here. We only used a single robot in our experiments (see the description in the text).
Fig. 6. The large stereoscopic projection system with the virtual version of the iSpace is shown here. The actual iSpace is in Japan, the virtual projection (on this picture) takes place in Hungary. We controlled the robot from the virtual environment in Hungary, and displayed the actual robot position as measured in Japan by the positioning system there. V. PREDICTION OF MOBILE ROBOT POSITION The network connection between the two laboratories has an average round-trip latency of 300 milliseconds. In order to compensate for the delay, the virtual robot’s position must be predicted based on the parameters of its motion. Without this compensation, the network delay would result in an incoherent and fuzzy image of the virtual robot in Hungary. Before this compensation takes place, we needed smooth positional data from the robot’s positioning system in order to be able to predict its position. As stated earlier, the use of EKF on ultrasonic measurement and wheel encoder data reduced the measurement noise and thus the positional error. Our implementation of the EKF is described in detail in . The combination of different sensor data (ultrasonic positioning system, laser-range finders and the robot’s wheel encoders) is required in order to reduce position measurement errors. Raw sensor data is collected and processed (applying EKF) by iSpace components and is sent to the main computer, which handles the communication over the internet. Network latency resulted in unacceptable visualization and inadequate control. This was mainly due to the uncertainty in the time delay, which was considerably high because of the wireless subnet in the real robot. The following state equations were introduced to describe the position and the orientation of the robot:
x v cos( ) , y = v sin( )
where x and y are the 2D coordinates and is the position-angle of the robot at any given time in the coordinate system that is tied to the floor of the room. Variable v denotes the speed of the robot and is the angular velocity around the center of the robot. Solving (1) in small discrete time intervals can predict the position of the robot, provided that the change in speed and angular velocity is not abrupt. In our case this latter condition will hold, since we will send commands in discrete time intervals to the robot over the internet, and predict the position of the local (virtual) robot during these short timeintervals when speed and angular velocity is not changing. The discrete form of (1) can be integrated with the above assumption. During time-step t both the speed and the angular velocity remains constant, therefore the position-angle for (that is in [0, t]) can be calculated as:
(t + ) = (t) + (t) = (t) + .
be the largest index for which
cos( (t + ))d = . (3)
cos( (t) + )d =
= x(t) +
The estimation is the following:
Upon receiving a new synchronisation message the simulator updates the state by calculating the estimation for and calculating a weighted average:
= x(t) +
from (5). Let
Now, integrating (1) is simple: x(t + t) = x(t) +
The estimation of the position at time
(sin( (t) + t) sin( (t)))
A similar solution can be written for y(t+t), and we arrive to: x(t) + (sin( (t) + t) sin( (t))) x(t + t)
y(t) y(t + t) (cos( (t) + t) cos( (t))) =
(t + t)
(t) + t
In current implementation this weight is set to . The commands, which are no longer affecting the prediction, are removed from the list and the estimated affection time of the first command is modified to be equal with the time of the synchronisation. The proposed concept is summarized in Fig. 7.
Based on (4), a position predictor can be constructed, which predicts the robot’s real response to the given commands. The simulator uses the last synchronisation message where is the time of measurement and the issued commands after
ordered by the estimated time of execution command
is set to be equal to
predictor is given by
. For the first
. The state of the . During the
estimation the predictor calculates the effects of the commands: Fig. 7. Concept of predictor
Two different type of path have been executed in order to verify the behaviour of the predictor. In the following figures the real and virtual path are compared.
The first test case is shown in Fig. 8. This demonstrates the mobile robot following a variable curve. Based on the achieved results it can be clearly seen, that the virtual robot reacts faster to commands, than the real robot. This is caused by the fact, that the applied predictor does not take in account the acceleration model of the real robot and also because of the prediction of time delay of robot command execution. However the overall mean deviation is lower than 74.1 mm, which is an acceptable error rate in case of a very fast and simple predictor.
locations in the real and remote space in our experiment. It can be divided to two main tasks: sensing and synchronization of robot navigation in the real and remote space. Sensing consists of tracking the position of the robot in the space. The synchronization is based on motion prediction and we have shown that it is an attractive and easy solution for this task. The presented implementation is an example of how to introduce physical agents inside Intelligent Space by utilizing the distributed sensors present in the space. It gives a practical solution that uses easily available sensors and robust control algorithms. ACKNOWLEDGMENT The authors are highly indebted to Peter Korondi and Gabor Sziebig for their insights and support in the preparation of this manuscript.
Fig. 8. Test for variable curve following of mobile robot In the second test case the robot was driven to predefined positions (-1, -0.9) and (-1.2, -1.7) and (0.1, -0.6) and (0.6, -0.8) and (1, -1.4), where the first coordinate is the and the second is respectively. The outcome of the test case can be seen in Fig. 9.
Fig. 9. Test for destination approach of mobile robot
It can be observed, that in this case the predictor works better which is the result of the mainly linear motion. The mean deviation in this case is lower than 44 mm. The current system setup only considers delays in one direction; the robot is visible at the estimated current position in real-time, but the control is still delayed according to the image and the one way latency.
VI. CONCLUSION In this paper the concept of extended Intelligent Space, stereoscopic virtual 3D copy, was presented. A mobile robot was guided simultaneously to desired
P. Korondi, and H. Hashimoto, ”Intelligent space, as an integrated intelligent system”, Proc. of the International Conference on Electrical Drives and Power Electronics, pp. 24-31. 2003. J. Krumm, S. Harris, B. Meyers, B. Brumitt, M. Hale and S. Shafer, "Multi-Camera Multi-Person Tracking for EasyLiving", Proceedings of the 3rd IEEE International Workshop on Visual Surveillance(VS), pp. 3-10, 2000 L. Rudolph, "Project Oxygen: Pervasive, Human-Centric Computing - An Initial Experience", Proceedings of the 13th International Conference on Advanced Information Systems Engineering (CAiSE), 2001 B. Johanson, A. Fox and T. Winograd, "The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms", in IEEE Pervasive Computing 1/2, pp. 67-74, 2002 T. Hori, A. Kaneko, K. Nagashima, Y. Nishida, H. Aizawa and H. Mizoguchi, "Delivery Task by a Humanoid Robot in the Sensorized Environment" in Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC), pp. 2930-2935, 2004 J.H. Kim, Y.D. Kim and K.H. Lee, "The Third Generation of Robotics: Ubiquitous Robot", 2nd International Conference on Autonomous Robots and Agents, pp. 1-7, 2004 B.K. Kim, N. Tomokuni, K. Ohara, T. Tanikawa, K. Ohba and S. Hirai, "Ubiquitous Localization and Mapping for Robots with Ambient Intelligence", in Proceedings of the IEEE international Conference on Intelligent Robots and Systems (IROS), pp. 48094814, 2006 G. Sziebig, A. Gaudia, P. Korondi, N. Ando, and B. Solvang, “Robot vision for rt-middleware framework,” in Proc. IEEE Instrumentation and Measurement Technology Conference (IMTC’07), Warsaw, Poland, pp. 1–6, 2007 D. Brscic, “Mobile robot as physical agent of intelligent space”, Journal of Computing and Information Technology, 2008, submitted for publication. J. H. Lee, H. Hashimoto, “Intelligent Space - Concept and Contents”, Advanced Robotics, 16/3 (2002), 265-280. D. S. Cohen, “James Cameron supercharges 3D”, www.variety.com web exclusive, Apr. 10, 2008. “Bic Camera, the world's first stereoscopic 3D 46 LCD TV sales”, www.watch.impress.co.jp, Apr. 12, 2008. H. Jorke, and M. Fritz, “Infitec – A new stereoscopic visualization tool by wavelength multiplex imaging”, Proc. of Electronic Displays, Wiesbaden, 2003. C. Cruz-Neira, D. Sandin, T. DeFanti, R. Kenyon, and J. Hart, "The CAVE: Audio visual experience automatic virtual environment". Communications of the ACM 35 (6), pp. 65-72, 1992. Object-Oriented Graphics Rendering Engine, OGRE 3D, http://www.ogre3d.org OpenGL handbook: http://www.opengl.org D. Brscic, T. Sasaki, and H. Hashimoto, “Implementation of Mobile Robot Control in Intelligent Space”, SICE-ICASTE International Joint Conference, 2006