topical media & game development
[] readme course(s) preface I 1 2 II 3 4 III 5 6 7 IV 8 9 10 V 11 12 afterthought(s) appendix reference(s) example(s) resource(s) _

talk show tell print

[|] / create / web3d / flex / video

topical media

readme

view(s) / ximpel / tag(s) / create(s) / present(s) / lecture(s) / content(s) / dolphin / study / clock / page(s) / media / bounce / multimedia @ VU


contents

present(s) / help(s) / code(s) / pattern(s) / tutorial(s) / list(s) / video(s) / tube(s) / web3d / resource(s) / ? / content(s)

copyright

This material is copyrighted. You may not copy or print any of this material without explicit permission of the author or the publisher.

formats

The notes are available

  • in PDF format
  • online in HTML
  • as slides in both dynamic HTML and VRML
  • on CDROM

slides

Slides are segments or slices of the text that may be displayed independently, or in sequence, with enlarged font.

Slides are available both in dynamic HTML/javascript presentation format, as well as in a home-grown VRML format.

PDF (reading and printing)

Go to print.

A PDF reader must be installed on your system. Such a reader can be freely obtained at www.adobe.com.

Alternatively, click on the sign at the top right for a PDF version of that particular chapter or section.

online version

Click on [], in the topleft of the navigation area, for a contents listing.

presentation

Click on the at the index page,or any of the other pages, for a dynamic HTML version of the slides.

See DejaVU Online (Readers Guide) for an explanation of the navigation options. The principle is simple, click on the right yellow sidebar, until the presentation is finished.

For a VRML version, click on the sign. These links require the blaxxun Contact 3D plugin. Hit the spacebar to advance in the sequence. The backspace takes you one back.

Also included is an experimental annotated VRML version, based on the DLP+VRML platform described in chapter 7. Click on the logo, to activate the presentation. After waiting about 20 seconds, two avatars will appear that comment on the presentation. Interaction proceeds as for the VRML presentation.

other links

In resources links to other sites as well as online copies of external material are given.

platform

The material has been tested with Explorer 6.0 and Netscape 4.7, with the blaxxun Contact 3D plugin 5.1. The annotated versions are developed using the DLP+VRML platform. These run only with Explorer 6.0.

The blaxxun Contact 3D VRML browser you need for this may be freely obtained from www.blaxxun.com.

print / present / tag(s)

preface -- topical media

This book provides a concise and comprehensive introduction to multimedia. It arose out of the need for material with a strong academic component, that is material related to scientific research.

Indeed, studying multimedia is not only fun. Compare it with obtaining a driver license. Before you are allowed to drive on the highway, you have to take a theory exam. So why not take such an exam before entering the multimedia circus.

Don't complain, and take the exam. After all it makes you aware of the rules governing the (broadband) digital highway.

themes and variations

 So, who is this book meant for? It is meant for the student or reader who is looking for a quick introduction to the main topics in multimedia. The twelve chapters provide a concise overview of the themes and trends in current multimedia practice and research.

The themes and variations addressed in this book may be summarized as follows.

themes and variations


  • digital convergence -- all for one, one for all
  • broadband communication -- entertainment
  • multimedia information retrieval -- as an afterthought?
  • multimedia and game application(s) -- from design to development

To explain in somewhat more detail, digital convergence may be characterized as the coming together of data (including audio, video and information) in a possible multitude of platforms, to which these data are delivered by a variety of (broadband) communication channels. In fact, the increasingly powerful communication infrastructure due to the popularity of the Internet and the World Wide Web, leads to an almost universally accessible multimedia (information) repository, for which (unfortunately) the notion of (multimedia) information retrieval seems to have occurred only as an afterthought. Digital content design is only one step in the process of multimedia application development. Important issues in multimedia application development are, apart from project maanagement, data representation, navigation, presentation and usability.

An underlying thought that motivated the writing of this book is that somehow the gap between authoring and retrieval should be bridged.

In other words, either by developing the technology for extracting features or attributes from multimedia objects, or by applying content annotation for such objects, multimedia information retrieval should be considered as a necessary asset to make a multimedia web an effective information repository. In multimedia applications, such as the digital dossier we introduce in chapter 10, the data representation must accomodate meta-information, to support effective navigation and search.

Another line of thought, that became more clear during the writing of the book is concerned with the aesthetics of (interactive) applications. You will find more on this in chapters 11 and 12, that deal with game development.

what do you need to learn

When taking up multimedia as a subject of study, you may ask yourself what you need to know and learn about it. In general, what this book presents is

what do we have to learn?
  • concepts,
  • facts,
  • history,
  • applications
  • standards
  • technology
  • multimedia and games

Let me be frank with you. There is too much information to be digested in a first course. Nevertheless, after studying this book you will have an introduction to multimedia that should be viable for the rest of your (academic) career.

Now, don't hesitate, put yourself to the test and check which phrases and acronyms you are familiar with in the lists given for the subjects of digital convergence, broadband communication and information retrieval.

digital convergence

  • concepts -- digital revolution
  • facts -- from the entertainment industry
  • history -- from Pong to Big Brother
  • applications -- infotainment
  • standards -- MPEG, RM3D, SMIL
  • technology -- TV, PC, DVD

...

How did you succeed thus far? If you did well, try the second round and test yourself in what detail you have have knowledge about technologies mentioned.

broadband communication

  • concepts -- Quality of Service
  • facts -- compression is needed
  • history -- the internet
  • applications -- entertainment and communication
  • standards -- HTTP, TCP/IP, RTP
  • technology -- cable, (X)DSL

...

Finally, check to what extent you master the vocabulary of multimedia information retrieval.

multimedia information retrieval

  • concepts -- features, precision, recall
  • facts -- the problem is utterly complex
  • history -- from text to multimedia
  • applications -- digital libraries
  • standards <-- distance metrics
  • technology -- indexing & algorithms

If you are working online, you may click back to the text in the book that explains these notions. Just to make sure whether your impression of familiarity was justified.

assignment(s)

I strongly believe that practical work is necessary, also for academics, to get a good grasp on multimedia and game development. Even if your interest is purely intellectual, it pays off to make your virtual hands dirty and indulge in making a compelling presentation.

As an assignment, consider making a presentation that offers an

assignment

Annotated Tour in Amsterdam

Amsterdam is the place where I live, and where our students take their courses. You may find it more convenient or natural to replace Amsterdam with a location of your choice.

Online, you will find an elaborated version of the assignment, including an extended description, a working plan, deliverables and hints. In essence though, the intent of the assignment is to make a compelling, not to say artistic, presentation, and to explore the realm of multimedia rethorics.

As a tool you may choose, for example, Flash or the flex 2 SDK, which is freely available.

examination

Despite the fact that some consider the practical aspects of multimedia to be exclusively relevant, the intellectual aspects of multimedia should not be ignored.

Consider the following question, which is directly related to one of the themes underlying this book, that is the complementarity of authoring and retrieval:

multimedia

 This question can only be answered when the student has a sufficient level of experience, insight and knowledge of the field, and is able to relate theory and practice.

Each chapter contains a brief list of questions that may be used as a checklist, to see if you have sufficient knowledge of a particular area. These questions may also be used to prepare exams!

The questions are meant to test for insight, that is the ability to discuss a somewhat broader theme, and knowledge of concepts and technology, covering definitions, applications, historical facts, as well as the technological infrastructure enabling the deployment of multimedia applications.

In addition to the regular material, the book also contains a number of examples and sections indicating research directions. These sections are not meant to be part of the exam, but might provide the student with suggestions for projects or further research. Moreover, both the discussions in the research directions and the material in the appendices presents a vision on what multimedia should be. In effect, I have a strong preference for a programmatic approach to (intelligent) multimedia, as outlined in appendix E. Nevertheless, the bulk of the (regular) material is relevent also for readers with a rather different opinion on what consitutes the essence of multimedia.

how to use this book

The intended audience for this book is

intended audience(s)


The course notes were explicitly written for first year Computer Science and Information Science students. The Information Science students are expected to choose the specialisation Multimedia and Culture, a curriculum provided by the department of Mathematics and Computer Science of the Faculty of Sciences of the VU University Amsterdam. The course has a practical part and a theoretical part, which in combination takes 2-4 weeks, full time study. The book covers the theoretical part. The online version gives a skeleton assignment that may adapted by the one responsible for the course. The online version contains all the material needed for giving a multimedia course, that is

multimedia course


  • presentations for all chapters, including the preface in dynamic HTML slides
  • presentable versions of the MPEG-4 standard, and other relevant material
  • possible exam questions, with back links into the text for quick learning and review
  • seven sample lectures, with additional explanation for the instructor
One additional remark may be made. This is (so to speak) 'a book with an attitude'. It is slightly authorative and directive towards the students, telling them to learn the facts and 'do the exam'. Some students take refuge to learning the 'keywords and phrases'. They are even helped in this respect, since the text uses a 'graphic' layout to emphasize important points, and to allow for a quick recognition of chunks of relevant material.

the artwork

 Although a book about multimedia does not need to be a multimedia artefact itself, it seemed better to include illustrations, to avoid the impression of a 'dry' book. Since I did not want to include any redundant diagrams or pictures, I decided to use a personal selection from the history of visual design, games, computer art and video art, not only to spice up the book but also to give the reader a collection of interesting samples. Each chapter starts with illustrations setting the visual theme of the chapter. All other illustrations are, in one way or another, related to the examples or the text of that chapter. Brief comments about the artwork, and an explanation of the visual theme, can be found at the end of each chapter.

about the author

 At some point you may wonder whether the author is qualified or authorized to write about a particular subject and, in this particular case, to publish a book about such an elusive notion as multimedia.

Let me give you some personal history. Way back in the seventies, I did a degree in painting at the Gerrit Rietveld Academy in Amsterdam. At the same time, I did a master's philosophy, where I graduated in the field of aesthetics on a comparative study on theories of imagination and creativity, reading writers such as Kant, Husserl and Sartre. Then I got an interest in computer music, after listening to a concert of Xenakis in Paris, and started to work on a PDP-15 (with 4K of memory) at the Institute of Sonology in Utrecht. Leaving all philosophy and traditional art behind, I learned programming, studied AI and theoretical computer science. Some eight years later, I obtained my Ph.D. in computer science and started my academic career. After working in software engineering, and in particular object-oriented software development, I was asked, at the end of the millenium, to set up a collection of multimedia courses, since by then multimedia was coming in vogue as an academic subject. These courses, which include the introduction multimedia, Web3D authoring, intelligent virtual environments, a multimedia casus, and recently also visual design, are reflected in this book.

about the book

What started as a (not so) gentle introduction to multimedia, has grown into a rich (at times somewhat idiosyncratic) collection of topical material about multimedia and game development. Borrowing a phrase from the politics of the seventies, at some point, apparently, the professional became personal, and the personal professional. Nevertheless, the book may still be read as an introduction. It is written in a concise and compact manner, supported by the slides format, which allows for presentation of the material in class, and is illustrated by a variety of images, taken from the arts, design, and multimedia and game projects.

The book consists of the following parts:

part(s)


  • part i -- digital convergence
  • part ii -- delivery & presentation
  • part iii -- multimedia information retrieval
  • part iv -- applications & tools
  • part v -- game development
Actually, as will become apparent when reading, the book is the result of a series of revisions. Since I started writing the book, the vision of digital convergence has become a reality, game research has become a respectable academic discipline, and attention has shifted from new media to cross media. Moreover, the role of media in our society is subject to change as well. With the Web 2.0, our information society is no loner passive, but part of a participatory culture in which users contribute and ultimately control content. And these developments are not likely to stop or slow down.

As concerns the history of this book, after writing chapters 1-8 (minus 4, which was added later), I extended the manuscript with with chapters 4, 9 and 10. After a thorough revision, I included the artwork, and then decided to subdivide the book in parts, adding chapter 11 and 12 in part v, on game development. The latest revision is the inclusion of another section with each chapter, discussing the latest development(s). Not visible in the printed book are the numerous technical examples, including ajax, flex and wiki experiments, that may help the reader in his/her exploration(s). Over the last year(s), it has become evident for me that the only viable method of learning multimedia is exploratory development. That is not to say that the theoretical material is superfluous. On the contrary. But only in the actual context of developing meaningful applications, meaningful for the individual that is, does the necessity of theory become an obvious truth.

acknowledgements

This book is the result of developing the course notes for an introduction to multimedia for first year Computer Science and Information Science students. Hence, first of all, I like to thank the students that had to endure all the rough drafts of this material, and perhaps not to forget my experiment(s) with the presentation format of it.

Further I like to thank Harrie van der Lubbe and Sander Lammers for developing the manual for Director and their support in developing the practical assignment. Also, I like to thank Martin Kersten from CWI for allowing me to join his Multimedia Database Systems research group as a guest for a period of about two years, and Alex van Ballegooij for his active involvement in the RIF project and his coding effort for the slide PROTOs, used to produce the presentation slides for this book and described in appendix B. Also from CWI, I like to thank Lloyd Ruttledge, Lynda Hardman and Jacco van Ossenbruggen, for their effort in thinking about the multimedia course in its initial stages, and Lloyd and Jacco for their involvement in some of the practical work, and Jacco in particular for his knowledge of hypermedia systems that he shared with me during the period that he was my Ph.D. student. From CWI, I like to thank Zsofi Ruttkay for her general interest in 'my projects'. From the VU, I like to thank Andy Tanenbaum for allowing me to use his material on digital video, Gerrit van der Veer for taking the initiative for Multimedia and Culture, Zhisheng Huang for his excellent contributions to the WASP project, Johan Hoorn for our spirited cooperation, and Claire Dormann, for our discussions on the direction the Multimedia and Culture curriculum should take, and for sharing her thoughts on persuasive technology with me. I also like to thank Tatja Scholte and IJsbrand Hummelen from ICN (Netherlands Institute for Cultural Heritage) for their contributions to the multimedia casus, and Gaby Weijers and Bart Rutten from Montivideo for their cooperation and all the video material they so generously provided. Thanks are due to Mark Veldhuijzen van Zanten, Jaap Stahlie, Peter van Kessel, Federico Campanale and Katelijne Arts for providing me with material. Special thanks goes to the student members of the 2004 autumn multimedia casus group, for their collective work on the abramovic dossier, and to Rutger van Dijk for rekindling my interest in C++/DirectX programming by his youthful enthousiasm. I also should not forget two students from the Computer Science master Multimedia, who were exceptional in their dedication and skills, Winoe Bhikharie, who acted (among others) as the manager in the development of the VU-Life game described in 11.2, and Marco Bouterse, who excelled in both DirectX and Half-life shader programming. Also I need to mention Marek van de Watering, a student from Multimedia and Culture, who often surprised me by his sincerity and sensitivity, and Hugo Huurdeman, who convinced me of the viability of interactive (flash) video and who proved to be an extremely dedicated and creative worker. No doubt, I owe thanks to Gaynor Redvers-Mullon who made a serious attempt to encourage me to get the best out of this manuscript, even though at some point I decided to do it in my own way. Further, I also wish to thank Dhaval Vyas, at that time PhD. student at VU, for his interest in my projects and for sharing his thoughts on the panorama project, znd Zeljko Obrenovic, postdoc at CWI, for his enthusiasm, his technical skills and his willingness to assist me with my teaching. I thank Pier Vellinga, who I googled as the spin-doctor of climate change, for involving me in Clima Futura, and not in the least for the pleasant cameraderie we enjoyed during the effort to obtain the dutch national prize for academic communication, and Frans Feldberg, for involving me in the VU @ Second Life effort, an equally refreshing interdisciplinary partnership, with which we got into the 8 o'clock nation TV news, as planned! of course, I should not forget the guys from the communication department of the VU, Egon Compter and Merlijn Draaisma, who perhaps somewhat overestimated my expertise, in their enthusiasm to involve me in their projects. Last but not least, I owe Yiwen Wang (王依文) my deepest gratitude for providing the motivation to continue the work, and to strive for beauty and pleasure, in a serious way.

Finally, I must mention that I owe much insight and material to (among others) the following books and articles:  [Subrahmanian (1998)],  [Forman and Saint John (2000)],  [Chang and Costabile (1997)],  [Ossenbruggen (2001)],  [Vasudev and Li (1997)],  [Klabbers (2006)],  [Grau (2003)],  [Kress and van Leeuwen (1996)], and not to forget  [Zielinski (2006)]. As in any intellectual endeavor, intellectual ancestry can hardly be praised enough. So let me briefly indicate, for each chapter, some of the sources that provided me with inspiration, insight and material:

references

  1.  [Forman and Saint John (2000)],  [Davenport (2000)],  [Jain (2000)].
  2.  [Chang and Costabile (1997)],  [Ossenbruggen (2001)],  [Klabbers (2006)].
  3.  [Vasudev and Li (1997)],  [Koenen (2000)],  [Visser and Eliens (2000)].
  4.  [Luna (2003)],  [Adams (2003)],  [Fernando and Kilgard (2003)]
  5.  [Subrahmanian (1998)],  [Baeza-Yates and Ribeiro-Neto (1999)].
  6.  [Subrahmanian (1998)],  [McNab et al. (1997)],  [Kersten et al. (1998)].
  7.  [Subrahmanian (1998)],  [Fluckiger (1995)],
  8.  [Fluckiger (1995)],  [Ballegooij and Eliens (2001)],  [Huang et al. (2002)].
  9.  [McCuskey (2002)],  [Bolter and Grusin (2000)],
  10.  [Chapman and Chapman (2004a)],  [Chapman and Chapman (2004b)],  [Klabbers (2006)],
  11.  [Sherrod (2006)],  [Grau (2003)],
  12.  [Juul (2005)],  [Arnheim (1957)],  [Hawkins (2005)],  [Kress and van Leeuwen (1996)].
The material in sections 4.3, 7.1, 7.3, chapter 8, sections 9.3 and 10.2, and section 11.2 reflect my own research efforts. The other material has all been diligently collected from (among others) the sources mentioned.

print / present / tag(s)

1

digital culture

life is becoming digital

learning objectives

After reading this chapter you should be able to define the notion of multimedia, recount the history of digital entertainment, explain the concept of digital convergence, discuss the future of cyberspace, and speculate about the commercial viability of mobile multimedia.

We live in the digital era,  [Negroponte (1995)]. We are surrounding ourselves with gadgets and we are consuming immense amounts of information, that is increasingly being delivered to us via the Internet. We play games, and we still watch (too much) television.

Some of us watch televion on our PCs, and may be even looking forward to watch television on their mobile phone. This is multimedia. For others, the PC is still a programmable machine. Being able to program it might earn you a living. Understanding multimedia, however, might even provide you with a better living.

In this chapter, we study what trends may currently be observed in the creation and delivery of multimedia information, and we explore what impact the digital revolution may have from a commercial perspective.

...

1

...

11

questions

1. digital convergence

concepts


technology


projects & further reading

As a project, consider the development of a Java-based mobile game using J2ME, see  [Morrison (2005)], or a web-based game using Visual Basic .NET, see  [Santos Lobao and Hatton (2003)].

You may further explore multiplatform game development, and find arguments to choose for either Java-based or managed code based implementations.

For further reading, I advice to have a look at the special issues of the Scientific American,  [American], and the CACM on the next 1000 years of computing,  [CACM (2001)], and, for getting an idea where this all leads to, Schneidermann's Leonardo's laptop,  [Shneiderman (2003)]. For Second Life, see  [Rymaszweski et al. (2007)].

the artwork

  1. photographs of art works by Marina Abramovic, Art must be beautiful, Blue period, Dissolution, Dozing consciousness, In between, with (pending) permission from Montevideo. See also section 10.2.
  2. medium, according to the Visual Thesaurus.
  3. fMRI Research on Virtual Reality Analgesia, see section 1.1.
  4. television and communication, according to the Visual Thesaurus.
  5. TV Today, exhibition at Montevideo, februari 2005.
  6. visible world -- taken from  [Rosenblum and Macedonia (2002)], see section 1.2.
  7. personal event database and personal gadgets, from Freeband project.
  8. Thomas Lips 1975, Thomas Lips 1993, from Marina Abramovic, with permission from Montevideo.
  9. scanlines from Woody Vasulka, 197x, with permission from the artist.
  10. VU @ SecondLife, taken from  [Eliens et al. (2007)].
  11. signs -- people,  [ van Rooijen (2003)], p. 254, 256.
The work of Marina Abramovic has a strong existential flavor. It has also served as the material for a case study in developing a digital artist dossiers, the abramovic dossier, discussed in section 10.2. The work of Woody Vasulka is of a more experimental character, and shows the joy of discovering the possibilities of the, at the time, new electronic and digital tools and materials.

print / present / tag(s)

2

hypermedia information spaces

everything must be intertwinkled

learning objectives

After reading this chapter you should be able to define information spaces in a precise manner, position the hypertextual capabilities of the web in a historical perspective, explain the difference between multimedia and hypermedia, and argue why computational support for narrative structure in multimedia applications is desirable.

However entertaining it might be presented to you, underlying every multimedia presentation there is an information space. That is to say, irrespective of the medium, there is a message. And being confronted with a message, we might want to inquire for more information.

In this chapter, we will define the notion of information space more precisely.

We will extend this definition to include information hyperspaces, by looking at the history of hypertext and hypermedia.

Finally, we will discuss visualisation as a means to present (abstract) information in a more intuitive way, and we will reflect on what is involved in creating compelling multimedia.

...

1

...

10

questions

2. information spaces

concepts


technology


projects & further reading

As a project, I suggest the development of a virtual tour in a city, museum or other interesting locatoion.

You may further explore the implementation of traversal within a context, taking into account the history of navigation when backtracking to a particular point, issues in hyperlinking and interaction in multimedia applications, and computational support for narratives.

For further reading I advice you to take a look at the history of hypermedia and the web, using online material from the W3C, or the history of media as accounted for in  [Briggs and Burke (2001)] and  [Bolter and Grusin (2000)].

the artwork

  1. book covers --  [Weishar (1998)],  [Eco (1994)],  [Burger (1981)],  [Kunst],  [Betsky (2004)]
  2. Federico Campanale -- Oxygen, fragments from video installation, 2004
  3. Vasarely --  [Diehl 1973].
  4. Vasarely --  [Diehl 1973].
  5. Vasarely --  [Diehl 1973].
  6. Federico Campanale -- Oxygen, more fragments.
  7. student work -- from introduction multimedia 2000.
  8. Rutger van Dijk -- mobius, interactive story, opening screen, see section 2.3.
  9. edgecodes -- screenshots, see section 2.3
  10. signs -- people,  [ van Rooijen (2003)], p. 244, 245.
The work of Vasarely has served as an example for many contemporary digital artists. It is playful, mat may be characterized also as formalist. The highly aesthetic video work of Federico Campanale who, as he told me was strongly influenced by vasarely in his early years, shows a similar combination of formalism and playfulness. The interactive story by Rutger van Dijk has a rather different atmosphere, it is highly romantic, with slick graphics. The musea sites are included to point to the existence of (an increasing number) of virtual tours.

print / present / tag(s)

3

codecs and standards

without compression delivery is virtually impossible

learning objectives

After reading this chapter you should be able to demonstrate the necessity of compression, to discuss criteria for the selection of codecs and mention some of the alternatives, to characterize the MPEG-4 and SMIL standards, to explain the difference between MPEG-4 and MPEG-2, and to speculate about the feasibility of a semantic multimedia web.

Without compression and decompression, digital information delivery would be virtually impossible. In this chapter we will take a more detailed look at compression and decompression. It contains the information that you may possibly need to decide on a suitable compression and decompression scheme (codec) for your future multimedia productions.

We will also discuss the standards that may govern the future (multimedia) Web, including MPEG-4, SMIL and RM3D. We will explore to what extent these standards allow us to realize the optimal multimedia platform, that is one that embodies digital convergence in its full potential. Finally, we will investigate how these ideas may ultimately lead to a (multimedia) semantic web.

...

1

...

18

questions

3. codecs and standards

concepts


technology


projects & further reading

As a project, you may think of implementing for example JPEG compression, following  [Li and Drew (2004)], or a SMIL-based application for cultural heritage.

You may further explore the technical issues on authoring DV material, using any of the Adobe, mentioned in appendix E. or compare

For further reading I advice you to take a look at the respective specifications of MPEG-4 and SMIL, and compare the functionality of MPEG-4 and SMIL-based presentation environments. An invaluable book dealing with the many technical aspects of compression and standards in  [Li and Drew (2004)].

the artwork

  1. costume designs -- photographed from Die Russchische Avantgarde und die Buhne 1890-1930
  2. theatre scene design, also from (above)
  3. dance Erica Russel,  [Wiedermann (2004)]
  4. MPEG-4 -- bits rates, from  [Koenen (2000)].
  5. MPEG-4 -- scene positioning, from  [Koenen (2000)].
  6. MPEG-4 -- up and downstream data, from  [Koenen (2000)].
  7. MPEG-4 -- left: scene graph; right: sprites, from  [Koenen (2000)].
  8. MPEG-4 -- syntax, from  [Koenen (2000)].
  9. MIT Media Lab web site.
  10. student work -- multimedia authoring I, dutch windmill.
  11. student work -- multimedia authoring I, Schröder house.
  12. student work -- multimedia authoring I, train station.
  13. animation -- Joan Gratch, from  [Wiedermann (2004)].
  14. animation -- Joan Gratch, from  [Wiedermann (2004)].
  15. animation -- Joan Gratch, from  [Wiedermann (2004)].
  16. animation -- Joan Gratch, from  [Wiedermann (2004)].
  17. Agneta and Frieda example.
  18. diagram (Clima Futura) game elements
  19. signs -- people,  [ van Rooijen (2003)], p. 246, 247.
Both the costume designs and theatre scene designs of the russian avantgarde movement are expressionist in nature. Yet, they show humanity and are in their own way very humorous. The dance animation by Erica Russell, using basic shapes and rhythms to express the movement of dance, is to some extent both solemn and equally humorous. The animations by Joan Gratch use morphing, to transform wellknown artworks into other equally wellknown artworks.

print / present / tag(s)

4

multimedia platforms

with DirectX 9 digital convergence becomes a reality

learning objectives

After reading this chapter you should be able to characterize the functionality of current multimedia platforms, to describe the capabilities of GPUs, to mention the components of the Microsoft DirectX 9 SDK, and to discuss how to integrate 3D and video.

Almost 15 years ago I bought my first multimedia PC, with Windows 3.1 Media Edition. This setup included a video capture card and a 4K baud modem. It was, if I remember well, a 100 Mhz machine, with 16 Mb memory and a 100 Mb disk. At that time, expensive as it was, the best I could afford. Some 4 years later, I acquired a Sun Sparc 1 multimedia workstation, with a video capture card and 3D hardware accelerator. It allowed for programming OpenGL in C++ with the GNU gcc compiler, and I could do live video texture mapping at a frame rate of about one per second. If you consider what is common nowadays, a 3Ghz machine with powerful GPU, 1 Gb of memory, a 1.5Mb cable or ADSL connection and over 100 Gb of disk space, you realize what progress has been made over the last 10 years.

In this chapter, we will look in more detail at the capability of current multimedia platforms, and we will explore the functionality of the Microsoft DirectX 0 platform. In the final section of this chapter, I will then report about the work I did with the DirectX 9 SDK to implement the ViP system, a presentation system that merges video and 3D.

...

1

...

14

questions

4. multimedia platforms

concepts


technology


projects & further reading

As a project, I suggest the development of shader programs using Rendermonkey or the Cg Toolkit, or a simple game in DirectX.

You may further explore the possibilities of platform independent integration of 3D and media, by studying for example OpenML. For further reading, among the many books about DirectX, I advice  [Luna (2003)],  [Adams (2003)] and  [Fay et al. (2004)].

the artwork

  1. dutch light -- photographs from documentary film Dutch Light.
  2. ViP -- screenshot, with morphing shader, see section 4.3.
  3. impasto -- examples, see section 4.1
  4. impasto -- after a painting of van Gogh, using Cg shaders,
  5. 3D vision, from  [Sullivan (2005)], see example(s) section 4.2.
  6. idem.
  7. photographs of DirectX and multimedia books, by the author.
  8. DirectX -- diagram from online documentation.
  9. ViP -- screenshot, with the news and animations.
  10. DirectX -- diagram from online documentation.
  11. DirectX -- diagram from online documentation.
  12. ViP -- screenshot, featuring Abramovic.
  13. Peter Frucht -- Reality of TV news, see section 4.3.
  14. Clima Futura -- architecture diagram.
  15. signs -- people,  [ van Rooijen (2003)], p. 248, 249.
The theme of the artwork of this chapter is realism. In the documentary dutch light, it was investigated whether the famous dutch light in 17th century painting really existed. The photographs shown here are a selection of shots that were taken on particular locations over a period of time. However, as an art historian formulated it in the documentary: dutch light is nothing but a bag of tricks shared by dutch 17th century painters. The examples from impasto demonstrated that, after all, realism is an arbitrary notion.

print / present / tag(s)

5

information retrieval

information retrieval is usually an afterthought

learning objectives

After reading this chapter you should be able to describe scenarios for information retrieval, to explain how content analysis for images can be done, to characterize similarity metrics, to define the notions of recall and precision, and to give an example of frequence tables, as used in text search.

Searching for information on the web is cumbersome. Given our experiences today, we may not even want to think about searching for multimedia information on the (multimedia) web.

Nevertheless, in this chapter we will briefly sketch one of the possible scenarios indicating the need for multimedia search. In fact, once we have the ability to search for multimedia information, many scenarios could be thought of.

As a start, we will look at two media types, images and documents. We will study search for images, because it teaches us important lessons about content analysis of media objects and what we may consider as being similar. Perhaps surprisingly, we will study text documents because, due to our familiarity with this media type, text documents allow us to determine what we may understand by effective search.

...

1

...

10

questions

5. information retrieval

concepts


technology


projects & further reading

As a project, you may implement simple image analysis algorithms that, for example, extract a color histogram, or detect the presence of a horizon-like edge.

You may further explore scenarios for information retrieval in the cultural heritage domain. and compare this with other applications of multimedia information retrieval, for example monitoring in hospitals.

For further reading I suggest to make yourself familiar with common techniques in information retrieval as described in  [Baeza-Yates and Ribeiro-Neto (1999)], and perhaps devote some time to studying image analisis,  [Gonzales and Wintz (1987)].

the artwork

  1. artworks -- ..., Miro, Dali, photographed from Kunstsammlung Nordrhein-Westfalen, see artwork 2.
  2. left Miro from  [Kunst], right: Karel Appel
  3. match of the day (1) -- Geert Mul
  4. match of the day (2) -- Geert Mul
  5. match of the day (3) -- Geert Mul
  6. mario ware -- taken from gammo/veronica.
  7. baten kaitos -- eternal ways and the lost ocean, taken from gammo/veronica.
  8. idem.
  9. PANORAMA -- screenshots from field test.
  10. signs -- people,  [ van Rooijen (2003)], p. 252, 253.
The art opening this chapter belongs to the tradition of 20th century art. It is playful, experimental, with strong existential implications, and it shows an amazing variety of styles.

The examples of match of the day by Geert Mul serve to illustrate the interplay between technology and art, and may also start you to think about what similarity is. Some illustrations from games are added to show the difference in styles.

print / present / tag(s)

6

content annotation

video annotation requires a logical approach to story telling

learning objectives

After reading this chapter you should be able to explain the difference between content and meta information, to mention relevant content parameters for audio, to characterize the requirements for video libraries, to define an annotation logic for video, and to discuss feature extraction in samples of musical material.

Current technology does not allow us to extract information automatically from arbitrary media objects. In these cases, at least for the time being, we need to assist search by annotating content with what is commonly referred to as meta-information.

In this chapter, we will look at two more media types, in particular audio and video. Studying audio, we will learn how we may combine feature extraction and meta-information to define a data model that allows for search. Studying video, on the other hand, will indicate the complexity of devising a knowledge representation scheme that captures the content of video fragments.

Concluding this chapter, we will discuss an architecture for feature extraction for arbitrary media objects.

...

1

...

13

questions

6. content annotation

concepts


technology


projects & further reading

As a project, think of implementing musical similarity matching, or developing an application retrieving video fragments using a simple annotation logic.

You may further explore the construction of media repositories, and finding a balance between automatic indexing, content search and meta information.

For further reading I advice you to google recent research on video analysis, and the online material on search engines.

the artwork

  1. works from  [Weishar (1998)]
  2. faces -- from www.alterfin.org, an interesting site with many surprising interactive toys in flash, javascript and html.
  3. mouth -- Annika Karlson Rixon, entitled A slight Acquaintance, taken from a theme article about the body in art and science, the Volkskrant, 24/03/05.
  4. story -- page from the comic book version of City of Glass,  [Auster (2004)], drawn in an almost tradional style.
  5. story -- frame from  [Auster (2004)].
  6. story -- frame from  [Auster (2004)].
  7. story -- frame from  [Auster (2004)].
  8. white on white -- typographical joke.
  9. modern art -- city of light (1968-69), Mario Merz, taken from  [Hummelen and Sill\'e (1999)].
  10. modern art -- Marocco (1972), Krijn Griezen, taken from  [Hummelen and Sill\'e (1999)].
  11. modern art -- Indestructable Object (1958), Man Ray, Blue, Green, Red I (1964-65), Ellsworth Kelly, Great American Nude (1960), T. Wesselman, taken from  [Hummelen and Sill\'e (1999)].
  12. signs -- sports,  [ van Rooijen (2003)], p. 272, 273.
Opening this chapter are examples of design of the 20th century, posters to announce a public event like a theatre play, a world fair, or a festival. In comparison to the art works of the previous chapter, these designs are more strongly expressive and more simple and clear in their message. Yet, they also show a wide variety of styles and rethorics to attract the attention of the audience. Both the faces and the mouth are examples of using body parts in contemporary art. The page of the comic book version of City of Glass, illustrates how the 'logic' of a story can be visualised. As an exercise, try to annoyaye the sequence of frames from the City of Glass can be described using the annotation logic you learned in this chapter. The modern art examples should interesting by themselves.

print / present / tag(s)

7

information system architecture

effective retrieval requires visual interfaces

learning objectives

After reading this chapter you should be able to dicuss the considerations that play a role in developing a multimedia information system, characterize an abstract multimedia data format, give examples of multimedia content queries, define the notion of virtual resources, and discuss the requirements for networked virtual environments.

From a system development perspective, a multimedia information system may be considered as a multimedia database, providing storage and retrieval facilities for media objects. Yet, rather than a solution this presents us with a problem, since there are many options to provide such storage facilities and equally many to support retrieval.

In this chapter, we will study the architectural issues involved in developing multimedia information systems, and we will introduce the notion of media abstraction to provide for a uniform approach to arbitrary media objects.

Finally, we will discuss the additional problems that networked multimedia confront us with.

...

1

...

10

questions

7. information system architecture

concepts


technology


projects & further reading

As a project, you may implement a multi-player game in which you may exchange pictures and videos, for example pictures and videos of celebrities.

Further you may explore the development of a data format for text, images and video with appropriate presentation parameters, including postioning on the screen and intermediate transitions.

For further reading you may study information system architecture patterns, nd explore the technical issues of constructing server based advanced multimedia applications in  [Li and Drew (2004)].

the artwork

  1. examples of dutch design, from  [Betsky (2004)].
  2. idem.
  3. screenshots -- from splinter cell: chaos theory, taken from Veronica/Gammo, a television program about games.
  4. screenshots -- respectively Sekken 5, Sims 2, and Super Monkey Ball, taken from insidegamer.nl.
  5. screenshots -- from Unreal Tournament, see section 7.3.
  6. idem.
  7. idem.
  8. resonance -- exhibition and performances, Montevideo, april 2005.
  9. CHIP -- property diagram connecting users.
  10. signs -- sports,  [ van Rooijen (2003)], p. 274, 275.
Opening this chapter are examples of dutch design, taken from the book False Flat, with the somewhat arrogant subtitle why is dutch design so good?. It is often noted that dutch design is original, functional and free from false traditionalism. Well, judge for yourself.

The screenshots from the various games are included as a preparation for chapter 9, where we discuss realism and immersion in games, and also because multiplayer games like Unreal Tournament have all the functionality a serious application would ever need.

print / present / tag(s)

8

virtual environments

augmented virtuality acts as an intelligent looking glass

learning objectives

After reading this chapter you should be able to characterize the notion of virtual context, discuss the issue of information retrieval in virtual environments, explain what is meant about intelligent multimedia and discuss the potential role of intelligent agents in multimedia applications.

From a user perspective, virtual environments offer the most advanced interface to multimedia information systems. Virtual environments involve the use of (high resolution) 3D graphics, intuitive interaction facilities and possibly support for multiple users.

In this chapter, we will explore the use of (desktop) virtual environments as an interface to (multimedia) information systems. We will discuss a number of prototype implementations illustrating, respectively, how paintings can be related to their context, how navigation may be seen as a suitable answer to a query, and how we can define intelligent agents that can interact with the information space. Take good notice, the use of virtual environments as an interface to information systems represents a major challenge for future research!

...

1

...

12

questions

8. virtual environments

concepts


technology


projects & further reading

As a project, I suggest the implementation of storytelling in virtual environments, with (possibly) an embodied agent as the narrator.

You may further explore or evaluate the role of agents in multimedia applications and virtual environments.

For further reading in (real) VR, I advice  [Sherman and Craig (2003)], and for gaining an understanding in story telling and applications you may try to get hold of the proceedins, of TIDSE 2003, and TIDSE 2004.

the artwork

  1. another series of dutch light.
  2. virtual context -- Dam Square, Amsterdam, see 8.1.
  3. VU Campus in VRML -- student project.
  4. CWI 3th floor, floormap and model, see 8.2..
  5. query -- on 3th floor of CWI.
  6. navigation -- on 3th floor of CWI.
  7. soccer game -- image from WASP project, see section 8.3.
  8. digital beauties -- taken from  [Wiedermann (2002)].
  9. digital beauties -- taken from  [Wiedermann (2002)].
  10. VU @ Second Life -- screenshots.
  11. signs -- sports,  [ van Rooijen (2003)], p. 276, 277.
Another sequence of dutch light, opening this chapter, is meant to make you wonder about realism. Is virtual reality less 'real'? With a reference to section 2.3, where I quoted  [Bolter and Grusin (2000)] on re-mediation, I may remark that the graphic style chosen for presenting the virtual environment strongly determines whether the environment is experienced as 'realistic'. In our culture this is generally a photorealistic style, as for example in the Mission Rehearsal Exercise discussed in the next chapter, section 9.2. The digital beauties are not only a pleasure to look at, but do also display a wide range of postures and styles.

print / present / tag(s)

9

digital content creation

post-modern design allows for sampling

learning objectives

After reading this chapter you should be able to mention some basic rules of digital content creation, discuss what criteria your portfolio should meet, describe how you would approach the design of a logo, explain the notion of user-centered design, and characterize the issues that play a role in dveloping multimedia for theatre.

Whether your ambition is to become a professional designer or not, also for students of information science and computer science, a course in visual design is a must, I think.

In this chapter, we will treat various aspects of digital content creation. The first section discusses how to approach visual design and gives a number of basic design assignments, that can be used to get experience with visual design. Section 2 discusses the issue of workflow and tools, and investigates how design fits in with the process of developing multimedia applications. In the final section, I will elaborate on a theatre project I was involved in, for which I had to develop an augmented reality application.

...

1

...

17

questions

9. digital content creation

concepts


technology


projects & further reading

As a project, you may develop a dialog engine for non-linear interactive story telling or a collage generator, that produces artworks from a collection of images.

You may further explore the various presentation platforms, and assess the tradeoffs with respect to the support they offer for authoring.

For further reading, I suggest to study interaction design patterns. It is also worthwhile to get some books on modern art, to gain some knowledge about art and design.

the artwork

  1. street logos -- images from  [Manco (2004)].
  2. photograph of oilpaint box.
  3. Mark Veldhuijzen van Zanten -- the six roles in their agency.
  4. Mark Veldhuijzen van Zanten -- to design for the salon, periodic lounge evenings in musea and art institutes.
  5. Geert Mul -- interactive multimedia installation.
  6. Geert Mul -- multimedia installation in dutch consulate in India.
  7. website of Institute of Creative Technologies, showing scenes from Mission rehearsal Exercise (MRE).
  8. street logos -- more images from  [Manco (2004)].
  9. website for Visual Sensations, a yearly VJ contest in the Netherlands, developed by the agency of mark Veldhuijzen van Zanten.
  10. Geert Mul -- Harbour Sound & Vision, 1999
  11. screenshots from virtual atelier of Marinus Boezem.
  12. left: don't spit, a chines poster against spitting during the SARS period, taken from dutch newspaper; right: filmteckarna,  [Wiedermann (2004)].
  13. sketches -- from filmteckarna,  [Wiedermann (2004)].
  14. sketches -- from filmteckarna,  [Wiedermann (2004)].
  15. sketches -- from filmteckarna,  [Wiedermann (2004)].
  16. game -- pizza boy, developed by Headland, see 9.3.
  17. signs -- health and safety,  [ van Rooijen (2003)], p. 258, 259
The artwork for this chapter is meant to emphasize context. The street logos opening this chapter, as well as the work of Mark veldhuijzen van Zanten and Geert Mul, must be experienced in a context to fully appreciate their meaning.

Also for the MRE application, it is the context, in this case the stress and anxiety of a war situation, that determines the impact. The photorealistic graphic style of MRE, wellknown by the trainees from other games, is meant to strengthen the experience of immersion. Notice that the street logos assume an almost iconic character.

print / present / tag(s)

10

application development

learn the craft, break through the magic of engineering

learning objectives

After reading this chapter you should be able to discuss the multimedia development process, to indicate the need for information system support in the cultural heritage domain, to characterize the notion of digital dossier, to provide solutions for navigating complex information spaces, and to discuss the data representation issues involved.

As you gather from reading this book, the field of multimedia is widely divergent. However, when you develop a multimedia application, you will find that all topics treated so far will become relevant. There will be a need to mix multiple media formats. You will have to find suitable codecs for your video. You will be asked whether search is possible. And, not the least important, you will have to balance navigation and presentation.

This chapter is based on the work we, that is my students, have been doing in the domain of cultural heritage. In the first section, we will introduce the notion of digital dossier and outline our general approach. We will then in section 2 look at some examples, and describe how we deploy concept graphs as a universal navigation tool for complex information spaces. Finally, in section 3, we will explore the options for presenting multimedia material and discuss the design issues as well as the technical issues that have arisen in the course of our work.

...

1

...

13

questions

10. application development

concepts


technology


projects & further reading

As a project, develop a data format for text, images and video in XML, and implement stylesheets in XSLT to convert the format for display, for example in HTML frames or using SMIL.

You may further explore the formulation of criteria for selecting software and tool support for developing multimedia applications.

For further reading I suggest, apart from the manuals and learning materials that come with your tools, to study example projects and in particular the workflow, that is the dependencies between stages in the production, as for example explained in  [McCuskey (2002)].

the artwork

  1. website of Montevideo Collection Catalogue. To avoid being parochial here, I should also mention similar institutes abroad, such as Electronic Arts Intermix from New York, USA, and LUX, from London, UK.
  2. website of INCCA.
  3. tangible virtual museum -- from  [Rosenblum and Macedonia (2005)], see section 10.1.
  4. digital dossier -- concept graph for abramovic dossier, see section 10.2.
  5. digital dossier -- presentation gadget in abramovic dossier, with video of Relation in Time, with Ulay.
  6. digital dossier -- installation Terra dea degli madre, as 3D model.
  7. conservator's studio -- Self-Portrait with Braid, see section 10.2
  8. diagram -- task world ontology,  [Welie et al. (1998)].
  9. tower of babel -- location where the event took place, see below.
  10. tower of babel -- projection of tower of babel project, see section 10.3, submitted by Katelijne Arts. The project is a concept of Katelijne Arts, Tineke Goemans, Franka van de Goor, Leidi Haaijer en Bert Vogels.
  11. tower of babel -- a view from the inside of the building.
  12. PANORAMA architecture -- from  [Si & Eliens (2007)].
  13. signs -- sports,  [ van Rooijen (2003)], p. 278, 279.
The artwork for this chapter is selected to emphasize variety and experiment. The collection of Montevideo contains a great number of works from the early history of video art, including the works of Nam June Paik and Bill Viola. Yet, despite the experimental flavor of these works, contemporary media art shows a strong sense of context, experience and communication, as demonstrated for example in the tower of babel project. The issues of preservation we dealt with in this chapter, may now, to conclude this chapter, be summarized as: how do we preserve the context of experience of contemporary media art?

print / present / tag(s)

11

game technology for serious applications

immersion does not require illusion but involvement

learning objectives

After reading this chapter you should have an idea how to approach the development of a moderately complex game, and you should also be able to discuss the notion of immersion and argue why using game technology is relevant for serious applications.

Game playing is fundamental to human life. Not only for entertainment, but also to acquire the necessary skills for survival. Game playing can take a variety of forms, but nowadays the dominant game paradigm is undoubtedly the interactive video game, to be played on a multimedia-enhanced PC or game console. Currently, games are being (re) discovered in the academic field, another serious areas of society, as excellent means for both the transfer of knowledge and, perhaps more importantly, for attitude change.

In this chapter we will look at the various issues in developing a game, and more specifically, in section 11.2 at requirements for a promotional game for our faculty and the issues that came up when giving a masterclass game development for high school students using this game. Finally, we will sketch the history of immersive systems, in particular panoramas, in section 11.3, and we will discuss how immersion is to be realized in a game context.

...

1

...

22

questions

11. game technology for serious applications

concepts


technology


projects & further reading

As a project, develop a non-violent game using the Source SDK. For example, you may develop an application that gives a community of users access their personal collections of photographs.

One interesting feature to explore is the use of narratives, that is a kind of guided tour that gives a user an overview of the collection of photographs by means of a story, taking (in other words) the user by the hand in navigating the gane space.

For further reading I suggest, apart from the manuals and learning materials that come with the Source SDK, books on game development such as  [Luna (2003)],  [Gee (2003)] and  [Klabbers (2006)].

the artwork

  1. digital beauties -- taken from  [Wiedermann (2002)].
  2. Masereel, social realist works
  3. Roy Lichtenstein, 1962
  4. Masereel, social realist works
  5. images from Samurai Romanesque, see section 1.3
  6. HalfLife 2 shader programming
  7. VU-Life 2 -- opening screen
  8. VU-Life 2 -- screenshots
  9. VU-Life 2 -- screenshots
  10. VU-Life 2 -- screenshots
  11. VU-Life 2 -- tools
  12. VU-Life 2 -- tools
  13. VU-Life 2 -- masterclass
  14. diagram AMICO core
  15. diagram AMICO applications
  16. Roy Lichtenstein, 1962, Stillives
  17. Monet, Nympheas
  18. Monet, Nympheas
  19. Monet, Nympheas
  20. Web 3Di -- diagram for Webvolution
  21. signs -- abstract,  [ van Rooijen (2003)], p. 146, 147.
The visual theme of this chapter is realism, on the one hand expressed by a choice from the work of the belgium artist Masereel, as well as in the screenshots of VU Life, which present a more or less realistic, that is recognizable rendering of our faculty. In the mean time, though, the restaurant has been rebuilt, making our virtual restaurant outdated, within a year after creating it. Also the cartooneske style of Roy Lichtenstein may considered to be strongly realistic, although in a slightly different sense. Finally, the nympheas of Monet represent a realistic epos of his famous garden, and in some sense an almost tragic attempt to express this in an artwork, which only adds to the realism of art, that is its vital role in our lives.

print / present / tag(s)

12

towards an aesthetics for interaction

experience is determined by meaning

learning objectives

After reading this chapter you should have an understanding of the model underlying game playing, and the role of narratives in interaction. Furthermore, you might have an idea of how to define aesthetic meaning in a cultural context, and apply your understanding to the creative development of meaningful interactive systems.

As in music, the meaning of interactive applications is determined, not only by its sensory appearance, but to a high extent by the structure and functionality of the application. This observation may, also, explain, why narratives become more and more important in current video games, namely in providing a meaningful context for possible user actions.

In this chapter, we take an interactive game-model extended with narrative functionality as a starting point to explore the aesthetics of interactive applications. In section 12.1, we will introduce a model for interactive video games, and in section 12.2 we will present a variety of rules for the construction of narratives in a game context. Finally, in section 12.3, we will characterize the notion of meaning from a traditional semiotics perspective, which we will then apply in the context of games and interactive multimedia applications.

...

1

...

13

questions

12. towards an aesthetics for interaction

concepts


technology


projects & further reading

As a project, explore the ways narratives may be constructed from a collection of images. Deploy the various editing facilities for providing flashbacks, flashforwards, and other (temporal) relations within storytelling.

You may implement this using flash, VRML, or even try to embed such a narration facility in a game level developed with the Delta3D or the Source SDK.

For further reading I suggest you to take a look at more theoretical material from media theory, such as  [Bolter and Grusin (2000)]. Also there is a large collection of books from MIT Media Press that is of relevance for our new visual culture.

the artwork

  1. einzelganger -- walking man of Alberto Giacommeti, taken from an aanouncement of the Ives Ensemble, Amsterdam.
  2. game component framework, from  [Björk & Holopainen (2005)].
  3. diagram MIME
  4. diagram experience as meaning
  5. Roy Lichtenstein, 1962, (a) Kiss II, (b) Masterpiece, (c) Forget it, forget me.
  6. edgecodes -- showing George Lucas and his editoroid.
  7. El Lissitzky, suprematist works
  8. El Lissitzky, suprematist works
  9. Roy Lichtenstein, 1999, Still lifes with brushstrokes
  10. Les Demoisselles dAvignon, Picasso, 1908, regarded as the start of Cubism, and Le Goutier, Jean Metzinger, 1911, often referred to as the Mona Lisa of Cubism.
  11. poster for exhibition of dutch china work.
  12. signs -- abstract,  [ van Rooijen (2003)], p. 228, 229.
The walking man is one of my favorate sculptures, for over a long time. It is also associated to the motto of part iv: a journey of a thousand miles begins with the first step . As an autobiographical note, the walking man, with einzelganger superposed (in translation loner), reflects the writing of topical media. In particular, the image put in sequence, reminds me of the repetitive complaints of my (former) superior at the faculty, who over and over again told me that I was always alone in my room, isolated, on an island. I must admit there is a truth in this, as I felt that the disciplines of software engineering and multimedia are widely divergent, and in that sense I was on my own. This book has undergone many rewritings, due partly to a clash between the expectations of others and my own vision on multimedia. And with a superior who emphasizes that he is "the boss", but has no intellectual authority nor any inspirational leadership whatsoever, at least not in the area of multimedia and gane development, there is really no other way than to go your own way. So I did it my way, indeed, quoting Paul Anka's song, made 'unforgettable' by Frank Sinatra.

In other words, after this brief autobiographical digression, the visual theme of this chapter on the aesthetics of interactive systems is on individual judgement, as exemplified among others by the suprematist works of El Lissitzky, the amplification of cartoons as art by Roy Lichtenstein, and the pioneers of Cubism. After all, individual judgement is what you need, when you wish to be involved in multimedia and/or game development.

print / present / tag(s)

afterthought(s)

The world of multimedia may be looked at in many ways. In fact, the phrase multimedia is too generic to be meaningful in any way. Nevertheless, multimedia has become a subject of interest for academia. This book has been written from an academic perspective. Let me clarify this perspective, to provide you with some context that might help you in understanding this book and use it more effectively in either education, research, or even your artistic endeavors.

As a starting point, let's look (again) at the media equation, quoted in the research directions of section 9.1:

media equation(s) 1/4


We regularly exploit the media equation for enjoyment by the willing suspension of our critical faculties. Theatre is the projection of a story through the window of a stage, and typically the audience gets immersed in the story as if it was real.

This suspension of our critical faculties seems opposed to what we are used to in academic practice. And, indeed, there is an often noted conflict between the arts and the sciences, a conflict that the introduction of multimedia in the academic curriculum cannot resolve.

If we try to delineate the 'meaning' of multimedia more precisely, we might come up with pseudo-equation such as the following.

multimedia equation(s) 2/4


multimedia = presentation + context

where presentation includes the sensory and aesthetic part and context everything else. Now, at the risk of getting too much involved in 'funny mathematics' we might define context by another series of pseudo-equations

multimedia equation(s) 3/4


  • context = convergence + information + architecture

where

multimedia equation(s) 4/4


  • convergence = data +platform + distribution
  • information = storage and retrieval
  • architecture = compression + components + connectivity
Clearly, and this is exactly what this exercise in funny mathematics intended to illustrate, this book is about the contextual aspects of multimedia. Contextual aspects that may be the subject of academic research.

Is there any hope to include the presentational or aesthetic aspects in the academic curriculum? Based on a thought experiment, that explored the possibility of algorithmic art and aesthetics,  [Eliens (1988)], I would say no. And as a matter of fact, I strongly disagree with a recipe-based approach to developing multimedia presentations, as seems to be popular in a number of the academic multimedia courses.

There is another shade of meaning that may be attributed to the notion of context, namely context of application. Evidently, multimedia has become a natural ingredient of almost any application you can think of. In 1998, I organized a course on multimedia for Ph.D. students, entitled Multimedia in Context. This course dealt with some of the issues in distributed multimedia and multimedia information retrieval, as well as applications in the publishing industry, travel advertisement and medical diagnosis. To announce the course, I used an image from medieval alchemy, see part I, and a phrase characterizing 'perfect solutions'.

perfect solutions


Much more than the art of turning base metals into gold, alchemy is a system of cosmic symbolism. The alchemist learns how to create within a sealed vessel a Model of the Universe in which the opposing complementary forces of Male and Female, Earth and Air, Fire and Water attain the perfect synthesis of which gold is the emblem.

Risking obscurity at this point, I wish to equate multimedia with alchemy, to emphasize that the engineering of multimedia is an art that takes a lifetime to master. Repeating the quote from section 9.1:

multimedia engineering


"engineering is the art of moulding materials we do not wholly understand ... in such a way that the community at large has no reason to suspect the extent of our ignorance."

multimedia in context

 Originally the book, that is chapters 1-7, were written for the Multimedia and Culture curriculum at the Vrije Universiteit, Amsterdam, that started in 2001. In particular, the book contains the course notes for the first year course introduction multimedia.

Then I constructed four follow-up courses:

multimedia courses


  • multimedia authoring -- Web3D/VRML
  • intelligent multimedia technology -- Virtual Environments
  • visual design -- digital content creation
  • multimedia casus -- digital dossier(s)
The first of these courses deals with the technology for creating 3D scenes and worlds see appendix B, whereas the second is about providing intelligent services in virtual environments, as discussed in chapter 8 and appendix E. In addition, Multimedia and Culture students are required to take a course in visual design and to work on a multimedia casus to bring what they learned into practice. The structure and content of these courses are reflected in chapters 9 and 10.

Due to faculty politics, the Multimedia and Culture curriculum was reduced to a minor in Information Science, which made it less appealing, both for students and staff, including me. Over time, all the course mentioned above were re-purposed for the newly created specialisation multimedia in Computer Science, which attracts more technically oriented students, with better skills for actual multimedia and game application development. Although not technical in nature, chapters 11 and 12 were written with these students in mind. And very likely, or hopefully, the specialisation multimedia will soon become multimedia and game development.

Faculty politics is not a nice topic, but unfortunately has an effect on daily life, even to the extent that I sometimes regret that I gave up the, within an academic environment, relatively safe and simple discipline of software engineering and object-oriented software development. The truth of the matter is that, whatever the reasons, multimedia and game development does not fit in well in the standard academic context of computer science. Although liked by students, it is certainly not well accepted, and for that matter understood, by the senior staff. But although anecdotes about the many conflicts about research directions and scientific merit would be interesting for those who like gossip, the sad fact is that the multidisciplinary background of multimedia and game development would require an intellectually and artistically more rich environment than a department of computer science can offer.

explorative development

From the perspective of research, the situation is not much better. It is my strong belief, right or wrong, that relevant research in the area of multimedia and game development requires explorative development, that is the design and implementation of prototype applications that embody the realization of an idea, as with our research on the digital dossier(s), an idea that includes technical as well as cultural and presentational aspects. But how hard it is to perform such multi-disciplinary research in an environment that is by tradition pre-dominantly mono-disciplinary.

Back to the book, apart from providing an introduction to a number of issues and research areas in the world of multimedia, this book also defines, in an implicit way, a research program that concerns the development and use of

virtual reality interfaces for multimedia information systems

All aspect covered in this book contribute, one way or another, to that (implicit) research program that may be classified under the heading of intelligent multimedia, of which a tentative definition is given in appendix C. And, admittedly, there are many aspects that are not covered, in particular those that are related to more advanced multimedia, virtual reality technology, and artificial inteligence.

the skill(s) of relevance

At this stage you may still wonder why I chose to name the book topical media & game development. Let me explain. The phrase topical, as an adjective, has the following meanings:

topical


  1. designed for involving local application (as an anesthetic),
  2. relating to, or arranged by, topics,
  3. referring to the topics of the day or place.
Although certainly not meant as an anesthetic, and even though it is arranged by, or at least refers to, topics, the intended meaning is due to the motivation to write a book that is relevant for the topics of (interest of) todays world. And, although it may not teach you the actual skills necessary to survive in todays world of multimedia and game development, it is certainly meant to help you in acquiring the skill of relevence, see  [Bruner (1972)], in this area, a skill that you will need to find your proper place and direction, anytime, anywhere.

Amsterdam, 2/9/2007

A. Eliëns

media @ VU

[] readme course(s) preface I 1 2 II 3 4 III 5 6 7 IV 8 9 10 V 11 12 afterthought(s) appendix reference(s) example(s) resource(s) _

(C) Æliens 23/08/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.