Program

Abstract: AI is revolutionizing not only the entire field of Computer Science, but nearly all fields of Science. However, while application contexts explode and LLMs display embarrassing cognitivequalities, the entire AI research field seems headed toward saturation of the fundamental ideas that have enabled today’s spectacular results of large companies. Is the infamous “AI winter” perhaps creeping into research? In this talk I argue that the time is ripe for a fundamental rethinking of AI methodologies with the purpose to migrate intelligence from the cloud to the growing global population of devices with on-board CPUs. To support learning schemes inspired by mechanisms found in nature, I propose developing intelligent systems within NARNIAN, a platform that enables social mechanisms and fosters learning processes over time, without the need for data storage.

Abstract: In this talk we will introduce four different types of foundation generative model, in the context of multimedia signal processing and control: autoregressive models, the class of diffusion, gradient flow and flow matching models, state space models, and basis-function approaches. We explain how and why each approach works, and what is necessary to train them. We then look at foundational representation learning models, why they work, and discuss how they can be used for multi-modal fusion. These pieces will be built together into a full tooling for foundational multimedia signal processing, along with demonstrations of multimedia foundation models in action. We will look at what is needed for combining signals, language, images, video and audio. Then in the second part we will look at the fundamental issues in using foundation models: domain shift and fine tuning, non-stationarity and adaptability, and finally computational cost and on-device implementation. Finally we will take a little time to review the state of the art development, and new understanding in the field.

Abstract: This talk will set the stage with a discussion of recent activity in artificial intelligence research. First, the basic transformer model will be explained. Next, the contrast between natural language and video processing will be highlighted. The talk will culminate in the discussion of a multimodal video transformer that utilizes semantically meaningful query and key features in learning context for action recognition.

Abstract: In this talk, the application of computer vision and AI to video analysis of the elementary classroom will be detailed. The talk will explain data collection, video scoring, illustration of results on a custom dashboard as well as a novel video coach for teachers. The use of multimodal inputs including video, audio, and text will be explored. Studies showing the efficacy of the AI methodology will be summarized. The talk will conclude with a discussion of open directions in AI research as well as lessons learned in the broad area of AI.

Abstract: In this talk, I will briefly review some recent advancements in machine learning security with a critical focus on the main factors that are hindering progress in this field. These include the lack of an underlying, systematic, and scalable framework to properly evaluate machine-learning models under adversarial and out-of-distribution scenarios, along with suitable tools for easing their debugging. The latter may be helpful in unveiling flaws in the evaluation process, as well as the presence of potential dataset biases and spurious features learned during training. I will finally report concrete examples of what our laboratory has been recently working on to enable a first step towards overcoming these limitations, in the context of malware detection and web security.

Abstract: In this talk I will provide an overview of the history of multimedia security and how it evolved from photographers wanting to protect images they posted on the Web to the explosion in generated media. I will describe early approaches to media content security including watermarking and steganography. Media security then transitioned from content protection into examining problems in media forensics where the goal is to determine if a media element was altered or manipulated. In recent years this has been extended to the analysis of generated media (e.g., “deep fakes”). I will discuss modern approaches to forensics analysis of images, video, and speech. Finally, I will talk about how the forensics community has circled back to look at watermarking as a way to potentially label generated content and the issues involved in deploying this approach. I will also describe several open research problems.

Abstract: Whereas biological systems continually learn throughout their lifetime, most artificial intelligence methods still operate in a rigid two-phase paradigm: a training phase followed by a deployment phase. When new data becomes available, these systems are typically retrained from scratch, discarding previous knowledge and requiring substantial computational resources. Continual learning proposes theory and methods to enable artificial intelligence to efficiently adapt to a non-i.i.d. data streams without forgetting previously acquired knowledge. In the first part of this talk, I will introduce the foundations of continual learning theory and discuss several methods to address catastrophic forgetting. In the second part of the talk, I will explore and present recent works on how continual learning can be applied in the foundation model era. I will conclude by discussing several open research questions.

Abstract: Large multimodal models map inputs from two or more modalities into an aligned representation space which facilitates cross-modal comparisons (e.g. text-to-image or image-to-text). A notable example of such multimodal models is OpenAI's CLIP, which was pre-trained on a web-scale collection of images and natural language. Through a contrastive alignment loss used during pre-training, CLIP has acquired a wide range of visual (textual) concepts that can be easily adapted to different tasks through textual (visual) prompts. In the first part of this talk I will introduce the fundamental concepts related to contrastive, multimodal pre-training and the wide variety of applications it enables -- applications made possibile due to the inter-modal alignment of multiple modalities. In the second part of this talk I will discuss the phenomenon known as the Modality Gap which occurs due to different input modalities mapping to different manifolds in the representation space. I will discuss how this misalignment manifests itself, demonstrate how the effects of misalignment depend on how the modality encoders are used, and point to ongoing work aiming to mitigate this misalignment inherent to contrastively-trained multimodal models.