Xingye Chen

Prompt learning has emerged as an efficient paradigm for adapting vision-language models (VLMs), e.g., CLIP, to downstream tasks without extensive fine-tuning. However, existing methods still suffer from insufficient alignment between textual and visual representations. To address this limitation, we propose a Concept-guided Attention Distillation framework for prompting (PromptCAD), which systematically integrates semantic knowledge from Large Language Models (LLMs) to enhance cross-modal alignment. Our framework operates through three key phases: 1) Concept Generation and Refinement: We first leverage LLMs to generate comprehensive category descriptions, from which representative semantic concepts are selected via clustering while eliminating irrelevant information. 2) Concept-guided Attention Distillation: A novel distillation mechanism aligns concept-guided attention maps (derived from LLM-generated semantics) with prompt-induced attention maps, guiding the adaptation process and improving text-visual modality alignment. 3) Multi-perspective Inference: During inference, a self-attention pooling strategy dynamically aggregates semantically relevant features from diverse local regions, enabling effective concept-guided reasoning. Extensive experiments on 11 datasets demonstrate that our method significantly outperforms existing prompt learning approaches across various recognition tasks. Our code is publicly available at https://github.com/Chenguoz/PromptCAD.

Few-Shot Class-Incremental Learning (FSCIL) poses a critical challenge in machine learning, requiring models to continuously integrate novel classes with limited samples while preserving knowledge of previously seen classes. While existing FSCIL approaches have demonstrated promising results, they still suffer from catastrophic forgetting and few-shot overfitting due to the challenge of balancing old knowledge retention with new knowledge acquisition. To address these challenges, we propose an innovative Semantic-Guided Global-Local Collaborative Prompt Learning (SGLC) framework. Built upon powerful pre-trained Vision-Language Models (VLMs), the framework first introduces a dual-alignment mechanism: globally aligning visual features with visual-textual prototypes and locally aligning multi-view visual features with local textual attribute features, which facilitates effective knowledge learning while preserving existing knowledge via frozen prototypes of previous classes. Furthermore, to alleviate overfitting, we incorporate Large Language Models (LLMs) to generate semantically rich textual descriptions, which simultaneously guide both global and local prompt learning through knowledge distillation. Extensive experiments on the miniImageNet, CIFAR-100, and CUB200 datasets demonstrate that SGLC performs favorably against the state-of-the-art methods.

In web data, advertising images are crucial for capturing user attention and improving advertising effectiveness. Most existing methods generate background for products primarily focus on the aesthetic quality, which may fail to achieve satisfactory online performance. To address this limitation, we explore the use of Multimodal Large Language Models (MLLMs) for generating advertising images by optimizing for Click-Through Rate (CTR) as the primary objective. Firstly, we build targeted pre-training tasks, and leverage a large-scale e-commerce multimodal dataset to equip MLLMs with initial capabilities for advertising image generation tasks. To further improve the CTR of generated images, we propose a novel reward model to fine-tune pre-trained MLLMs through Reinforcement Learning (RL), which can jointly utilize multimodal features and accurately reflect user click preferences. Meanwhile, a product-centric preference optimization strategy is developed to ensure that the generated background content aligns with the product characteristics after fine-tuning, enhancing the overall relevance and effectiveness of the advertising images. Extensive experiments have demonstrated that our method achieves state-of-the-art performance in both online and offline metrics. Our code and pre-trained models are publicly available at https://github.com/Chenguoz/CAIG.

Few-shot classification (FSC) is a challenging problem, which aims to identify novel classes with limited samples. Most existing methods employ vanilla transfer learning or episodic meta-training to learn a feature extractor, and then measure the similarity between the query image and the few support examples of novel classes. However, these approaches merely learn feature representations from individual images, overlooking the exploration of the interrelationships among images. This neglect can hinder the attainment of more discriminative feature representations, thus limiting the potential improvement of few-shot classification performance. To address this issue, we propose a Sample Relationship Exploration (SRE) module comprising the Sample-level Attention (SA), Explicit Guidance (EG) and Channel-wise Adaptive Fusion (CAF) components, to learn discriminative category-related features. Specifically, we first employ the SA component to explore the similarity relationships among samples and obtain aggregated features of similar samples. Furthermore, to enhance the robustness of these features, we introduce the EG component to explicitly guide the learning of sample relationships by providing an ideal affinity map among samples. Finally, the CAF component is adopted to perform weighted fusion of the original features and the aggregated features, yielding category-related embeddings. The proposed method is a plug-and-play module which can be embedded into both transfer learning and meta-learning based few-shot classification frameworks. Extensive experiments on benchmark datasets show that the proposed module can effectively improve the performance over baseline models, and also perform competitively against the state-of-the-art algorithms. The source code is available at https://github.com/Chenguoz/SRE.

Keypoints extraction from 3D objects is a fundamental task in point cloud processing. The ideal keypoints should be an ordered and well-aligned set of points that effectively reflect the shape and structure of the object. To this end, this paper proposes an unsupervised 3D point cloud keypoints generation network with the consideration of the probability distribution of keypoints and spatial distribution among keypoints. The network downsamples and groups the 3D point cloud, obtaining local features of the point cloud. The local features are leveraged to explicitly learn the mixture probability distribution of keypoint position. A composite loss function that comprehensively considers shape similarity, point importance, and geometric constraint is proposed to guide the network in generating keypoints with semantic consistency and regular spatial distribution. The experimental results and quantitative comparisons on the ShapeNet and KeypointNet datasets demonstrate that the proposed method achieves ordered, well-aligned, and robust keypoints generation for 3D point clouds. The source code of the proposed method is available at https://github.com/djzgroup/Keypoints.

Geometrical structures and the internal local region relationship, such as symmetry, regular array, junction, etc., are essential for understanding a 3D shape. This paper proposes a point cloud feature extraction network named PointSCNet, to capture the geometrical structure information and local region correlation information of a point cloud. The PointSCNet consists of three main modules: the space-filling curve-guided sampling module, the information fusion module, and the channel-spatial attention module. The space-filling curve-guided sampling module uses Z-order curve coding to sample points that contain geometrical correlation. The information fusion module uses a correlation tensor and a set of skip connections to fuse the structure and correlation information. The channel-spatial attention module enhances the representation of key points and crucial feature channels to refine the network. The proposed PointSCNet is evaluated on shape classification and part segmentation tasks. The experimental results demonstrate that the PointSCNet outperforms or is on par with state-of-the-art methods by learning the structure and correlation of point clouds effectively.