Invited Talks

2024

1. Online Optimization Meets Urban Transportation

   In Student Learning Hub Seminar Series, C2SMART Center, New York University, New York, NY, Nov 2024

   Abs Video Code

   Urban transportation networks are inherently complex and dynamic, characterized by intricate road connections and diverse network structures coupled with time-variant traffic demands and frequent traffic incidents. Hence, offline planning or designing alone cannot guarantee real-time operational control and management of urban transportation systems, which may fail when physical attacks, unforeseen conditions, or unanticipated use places the system outside the design envelope. A desired real-time operation mechanism must adapt to the dynamic environment and determine management decisions to be executed while a system is running; i.e., input data arising over time have to be processed, and decisions have to be made before all input data are known. Such a decision-making process falls within the realm of online optimization or online learning. Motivated by several intelligent transportation applications from our past research projects, this tutorial aims to provide a gentle introduction to online optimization methods with much emphasis on the intuitive insights and relevance to transportation applications. The tutorial starts with gradient descent algorithms in conventional convex optimization and then moves to online gradient descent in online optimization problems. Extending from the single-agent online optimization, we briefly touch upon multi-agent online learning and associated equilibrium convergence. We conclude the tutorial by discussing the openings and challenges when deploying online optimization in urban transportation systems.

2. Towards Agent-Based Autonomous Network Security

   In IEEE COMSOC TCCN Rising Star Symposium Series, Stevens Institute of Technology, Hoboken, NJ, Nov 2024

   Abs Video

   Security of cyber-physical network systems, such as 5G/6G communication networks, vehicular networks, and the Internet of Things, has become increasingly critical nowadays. Traditional security mechanisms rely primarily on manual operations, which can be slow, expensive, and ineffective in the face of the dynamic landscape of adversarial threats. This problem will only be exacerbated as attackers leverage artificial intelligence (AI) to automate their workflows. As a countermeasure, safeguarding critical network systems also calls for autonomous defensive operations that delegate security decisions to AI agents. This talk presents our agent-based framework for autonomous attack detection and response using reinforcement learning (RL) and large language models (LLM). To address conventional RL’s reactive nature, we propose a new RL paradigm, conjectural online RL (coRL), to equip the security agent with predictive power when dealing with the agent’s epistemic uncertainty over the attacker’s presence and actions. The intuition of coRL is to endogenize the epistemic uncertainty as part of the RL process: the agent maintains an internal world model as a conjecture of the uncertainty, and the learned conjecture produces valid predictions consistent with environment feedback induced by epistemic uncertainty. To mitigate the RL agent’s reliance on stylized modeling and textual data pre-processing, we further incorporate LLMs into the agentic framework to deliver end-to-end autonomous cyber operations. We finally conclude the talk by discussing the path ahead to building fully autonomous security agents.

3. Conjectural Online Learning in Asymmetric Information Stochastic Games

   In Systems Engineering Department Seminar Series, City University of Hongkong, Hong Kong, Oct 2024

   Abs

   Modern socio-technical network systems powered by artificial intelligence (AI) technologies feature sophisticated interactions among humans, AI agents, and system entities. Asymmetric information stochastic games (AISG) provide principled mathematical modeling for such interactions, leading to game-theoretical mechanisms for network management. However, existing computational and learning methods in asymmetric information stochastic games (AISG) are primarily offline without adaptability to online nonstationarity, which falls short of proactive intelligence for resilient network management. To address these limitations, we propose conjectural online learning (COL), an online learning framework for generic AISGs. COL uses a forecaster-actor-critic (FAC) architecture, where the forecaster conjectures the other agents’ strategies and system dynamics within a look-ahead horizon, representing the agent’s subjective (mis)perception of the AISG. Based on these subjective perceptions, COL employs online rollout (actor-critic) to improve the policy. Bayesian learning is then used to calibrate the conjectures using information feedback. We establish that the conjectures produced by COL are asymptotically consistent with the information feedback in the sense of a relaxed Bayesian consistency. We deploy COL in a nonstationary IT infrastructure digital twin, which delivers online adaptable defense against advanced persistent threats compared with benchmark reinforcement learning techniques.

4. Agent of Agents: Meta LLM-Agent for Autonomous Security Operations

   In NSF Workshop on Large Language Models for Network Security, Center for Cybersecurity, New York University, New York, NY, Oct 2024

   Abs HTML

   Today’s security operations are largely manual, slow, costly, and often ineffective. There is growing interest in moving toward more autonomous agents that can handle security operations with greater efficiency and lower operational costs. These agents can make decisions autonomously, adapt to new threats, and provide real-time automated responses. This shift is part of a broader transition from traditional rule-based engines to more sophisticated AI-driven engines, such as those powered by Reinforcement Learning (RL) and Large Language Models (LLMs). In this talk, we present a prototyping framework of he Meta Agent, or the Agent of Agents. A Meta Agent is essentially a mosaic of specialized agents, each focused on particular tasks, resulting in a cost-efficient and customizable agentic solution. By synthesizing insights from various agents, the Meta Agent can tackle complex tasks that no single agent could manage on its own. In addition to the Meta Agent concept, the integration of game theoretic methods and LLMs also provides a symbiotic framework for cybersecurity operations. From the bottom up, game models provide a strategic framework for analyzing and defining high-level goals. From the top down, LLMs can take these strategic commands and translate them into operational tactics, allowing the system to execute the desired actions effectively. This integration of strategy and operation offers a more holistic approach to managing security operations, ensuring that high-level decisions are carried out with precision at the tactical level.

5. Conjectural Online Learning with First-order Beliefs in Stochastic Games

   In , Coordinated Science Laboratory, University of Illinois Urbana-Champaign, Champaign, IL, Aug 2024

   Abs

   Existing computational methods for asymmetric information stochastic games (AISG) are primarily offline and can not adapt to equilibrium deviations. Further, current methods are limited to particular information structures to avoid belief hierarchies. Considering these limitations, we propose conjectural online learning (COL), an online learning method under generic information structures in AISGs. COL uses a forecaster-actor-critic architecture, where subjective forecasts are used to conjecture the opponents’ strategies within a lookahead horizon, and Bayesian learning is used to calibrate the conjectures. To adapt strategies to nonstationary environments based on information feedback, COL uses online rollout with cost function approximation (actor-critic). We prove that the conjectures produced by COL are asymptotically consistent with the information feedback in the sense of a relaxed Bayesian consistency. We also prove that the empirical strategy profile induced by COL converges to the Berk-Nash equilibrium, a solution concept characterizing rationality under subjectivity.

6. Automated Security Response Through Conjectural Online Learning under Information Asymmetry

   In , Autonomous Robotics and Control Lab, California Institute of Technology, Pasadena, CA, Jun 2024

   Abs

   We study automated security response for an IT infrastructure and formulate the interaction between an attacker and a defender as a partially observed, non-stationary game. We relax the standard assumption that the game model is correctly specified and consider that each player has a probabilistic conjecture about the model, which may be misspecified in the sense that the true model has probability 0. This formulation allows us to capture uncertainty and misconception about the infrastructure and the intents of the players. To learn effective game strategies online, we design Conjectural Online Learning (COL), a novel method where a player iteratively adapts its conjecture using Bayesian learning and updates its strategy through rollout. We prove that the conjectures converge to best fits, and we provide a bound on the performance improvement that rollout enables with a conjectured model. To characterize the steady state of the game, we propose a variant of the Berk-Nash equilibrium. We present COL through an advanced persistent threat use case. Testbed evaluations show that COL produces effective security strategies that adapt to a changing environment. We also find that COL enables faster convergence than current reinforcement learning techniques.

7. Multi-level Traffic-responsive Tilt Camera Surveillance through Predictive Correlated Online Learning

   In NYU Urban Research DAy, The Robert F. Wagner Graduate School of Public Service, New York University, New York, NY, Mar 2024

   Abs arXiv HTML Poster

   In urban traffic management, the primary challenge of dynamically and efficiently monitoring traffic conditions is compounded by the insufficient utilization of thousands of surveillance cameras along the intelligent transportation system. This paper introduces the multi-level Traffic-responsive Tilt Camera surveillance system (TTC-X), a novel framework designed for dynamic and efficient monitoring and management of traffic in urban networks. By leveraging widely deployed pan-tilt-cameras (PTCs), TTC-X overcomes the limitations of a fixed field of view in traditional surveillance systems by providing mobilized and 360-degree coverage. The innovation of TTC-X lies in the integration of advanced machine learning modules, including a detector-predictor-controller structure, with a novel Predictive Correlated Online Learning (PiCOL) methodology and the Spatial-Temporal Graph Predictor (STGP) for real-time traffic estimation and PTC control. The TTC-X is tested and evaluated under three experimental scenarios (e.g., maximum traffic flow capture, dynamic route planning, traffic state estimation) based on a simulation environment calibrated using real-world traffic data in Brooklyn, New York. The experimental results showed that TTC-X captured over 60% total number of vehicles at the network level, dynamically adjusted its route recommendation in reaction to unexpected full-lane closure events, and reconstructed link-level traffic states with best MAE less than 1.25 vehicle/hour. Demonstrating scalability, cost-efficiency, and adaptability, TTC-X emerges as a powerful solution for urban traffic management in both cyber-physical and real-world environments.

2022

1. On the Role of Information Structures in Multi-agent Learning

   In The 33th International Conference on Game Theory, Stony Brook Center for Game Theory, Stony Brook, NY, Jul 2022

   Abs HTML

2. Informationally Mosaic Reinforcement Learning

   In Special Session on Markov Descision Processes, SIAM 2022 Annual Meeting, Pittsburgh, David L. Lawrence Convetion Center, Pittsburgh, PA, Jul 2022

   Abs PDF

   Multi-agent Reinforcement learning (MARL) has shown encouraging successes in addressing the sequential decision-making problem of multiple autonomous agents within a dynamic environment. The key to its successes is that MARL enables agents to adjust strategies based on their perceptions of the surroundings and the feedback from the environment. We refer to the structure of feedbacks and perceptions as the information structure of MARL. To achieve a broader deployment in reality, MARL must be able to adapt agents to varying information structures. The issue of learning under unknown, dynamic, and generally amorphous information structures poses a great challenge to current MARL studies. To address it, we propose a novel framework, Informationally Mosaic Multi-Agent Reinforcement Learning (IMMARL), where agents with different information structures coordinate in an unprescribed way to explore and utilize constructive information from the environment. In particular, the agent’s exploration operates in a laissez-faire manner, that is, it voluntarily rewards others for discovering and sharing helpful information. The proposed framework brings up flexible interoperability, and increases the modularity in MARL systems. We introduce a novel metric, Value of Information (VoI), to quantify the importance of informational exploration during learning. We corroborate on the proposed IMMARL and VoI using experiments conducted in procedurally-generated benchmark environments.

2020

1. Multi-Agent Correlated Learning over Networks

   In Special Session on Game Theoretic Learning in Networks, INFORMS Annual Meeting, Online, Nov 2020

   Abs

   We study game-theoretic learning over networks, which is of great importance when studying multi-agent decision-making and it has been widely applied to various problems, ranging from smart grids, supply chain management and autonomous vehicles to cyber security and social networks analysis. Prior works on game-theoretic learning have not paid enough attention to the topology of the network as well as the relationship between local interactions within neighborhoods and overall correlations among all agents on the network. For example, centralized approaches rely on the global information, i.e., information about each individual in the networked system, which seems not practical or efficient in large and complex systems, whereas fully decentralized approaches, though more self-dependent, are unable to tell how the mutual influence among agents leads to a desired coordination. In this presentation, we shall show that how local correlations within neighborhoods result in a correlated learning over the whole network system, where the influence of each agent’s action is propagated through the underlying network, leading to coordinated behaviors of networked agents. More specifically, three aspects are included in this presentation. We first present the mathematical models for games over networks and introduce related game theory basics as well as necessary tools for analyzing strategic interactions among game players. Then we provide various interpretations of correlations over networks in different contexts, involving correlated equilibrium and correlated mechanism over networks. We point out those interpretations make such a game model quite flexible in the sense that it can be leveraged to model strategic interactions among participants of organizations or systems arising from different scenarios, such as economic or social institutions and wireless networks. Finally, we move to correlated learning over networks and we argue that it is not only a descriptive model for explaining how the correlations over networks are obtained from non-equilibrium dynamic behaviors but also sheds a new light on the prescriptive design of learning schemes for the networked agents so as to achieve certain desired correlations. We conclude this presentation by briefly discussing the confluence of game-theoretic learning, network systems and artificial intelligence (AI), which gives a promising route for the further development of game-theoretic learning in the era of modern AI.

2017

1. Directional Framelets and its Application in Medical Imaging

   In PIMS-AMI Workshop on Applied Harmonic Analysis, University of Alberta, Edmonton, Canada, Aug 2017

   Abs HTML

   A directional compactly supported d-dimensional Haar tight framelet is constructedsuch that all its high-pass filters in its underlying tight framelet filter bank haveonly two nonzero coefficients with opposite signs and they exhibit totally (3^d - 1)/2 directions in dimension d. Furthermore, applying the projection method to sucha tight framelet, a directional compactly supported box spline tight framelet withsimple geometric structure is built such that all the high-pass filters in its underlyingtight framelet filter bank have only two nonzero coefficients with opposite signs aswell. Moreover, such compactly supported box spline tight framelets can achievearbitrarily high numbers of directions by using refinable box splines with increasingsupports. Their application to pMRI with good performance is presented