Arijit Ray.|

Computer Science Ph.D. Student |

Projects CV Tutorials Github

Google Scholar
 

I wish to teach machines to help humans achieve more in the world.

About me and my research:

I am a Computer Vision Ph.D. Student at Boston University (BU), advised by Kate Saenko and Bryan Plummer. I collaborate closely with Ranjay Krishna from the University of Washington and am a student collaborator in the PRIOR Team at Allen AI. I received my M.S from Virginia Tech, advised by Devi Parikh.

Recently, I am interested in endowing AI models with spatial intelligence to help them effectively take actions in the 3D world. Since obtaining such annotations is expensive, I wish to teach models to imagine scenarios in 3D simulations and learn from them to generalize to the real world. I also enjoy creating ecosystems that encourage creative building. Hence, I am a member of the AI for Impact Venture Studio at MIT.

If you are interested to collaborate or just chat about research on multimodal models to help human creativity, say hi!

ai2

Student Collaborator, Allen AI (PRIOR), 2024
googleX

AI Resident, Mineral, Google X, 2023
meta_ai_fair

Research Intern, Facebook AI Research, 2022
sri

Computer Scientist, 2017-2021
blueriver

Deep Learning Intern, 2016, acquired by John Deere

Highlights:

  • 2024: Our paper on feedback-guided autonomous driving got accepted to CVPR 2024 as a highlight.
  • 2023: Our paper on evaluating and adapting models for compositional reasoning got accepted to NeurIPS.
  • 2023: One of my student's paper got accepted as an oral at an ICCV Workshop 2023.
  • 2023: I am excited to be spending my summer as an AI Resident at Google X (Mineral), working on adapting multimodal language models for object localization.
  • 2022: We started [AI+X] of BU and Harvard, a group where we brainstorm research/venture ideas on how AI can impact concurrent research areas.
  • 2022: I am excited to be spending my summer as a Research Scientist Intern at Meta (Facebook) AI (FAIR), working on the compositionality of large vision-language models.
  • 2019: I won runners-up at the SRI CVT Shark Tank Competition that supported my mini-project on understanding image-text content to reduce radicalization of opinions on social media.
  • 2017: The weed vs plant detection system I helped develop for precision fertilizing played a key part in the acquisition of Blue River Technologies by John Deere for 305 million USD.
  • 2016: My first paper got accepted to EMNLP.
  • 2014: Our UAV for helping locating natural disaster victims was featured in National News : Deccan Chronicle, Indian Express
  • 2013: I won a silver medal at SRM University Research Day for my white-paper presentation on an Electro-Mechanical Exoskeleton.
  • 2012: I won an Academic Merit Scholarship from SRM University that waives a part of my undergraduate tuition.

Projects/Publications:

Publications:

2024

Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna*, Kuo-Hao Zeng*, Kate Saenko*, SAT: Spatial Aptitude Training for Multimodal Language Models, [webpage] [arxiv], [data], [code coming soon!]

Tl; dr: Simulated spatial aptitude data (SAT) can improve spatial reasoning in real images for MLMs while maintaining pretraining commonsense. When instruction-tuned on SAT, LLaVA-13B matches some larger MLMs like GPT4-V and Gemini-3-1.0 in spatial reasoning.

Jimuyang Zhang, Zanming Huang, Arijit Ray, Eshed-Ohn Bar, FED: Feedback-Guided Autonomous Driving, CVPR 2024 (Highlight), [paper]

Tl; dr: MLMs can benefit autonomous driving by understanding natural language feedback and refining the next waypoint prediction.

2023

Dina Bashkirova, Arijit Ray, Rupayan Mallick, Sarah Adel Bargal, Jianming Zhang, Ranjay Krishna, Kate Saenko, Lasagna: Layered Score Distillation for Disentangled Object Relighting, [arxiv] [project page, data]

Tl; dr: Synthetcially generated examples are effective in teaching physics-aware edits like relighting if we use score-distillation to avoid overfitting.

Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan Plummer, Ranjay Krishna, Kate Saenko, Cola: A Benchmark for Compositional Text-to-image Retrieval, NeurIPS 2023, [arxiv] [project page, data]

Tl; dr: Tuning multimodal layers improve the unseen compositional reasoning ability in CLIP-style vision-language models the most over tuning other parts of the model.

Katherine Deng, Arijit Ray, Reuben Tan, Saadia Gabriel, Bryan A. Plummer, Kate Saenko, Socratis: Are Large Multimodal Models Emotionally Aware?, ICCV Workshops 2023 (oral), Workshop on Emotionally and Culturally Intelligent AI, [arxiv] [project page, data]

Tl; dr: A preliminary study showing MLMs seem to lack diverse perspectives for why different people may feel differently while viewing the same image-text content.

2022

Reuben Tan, Arijit Ray, Andrea Burns, Bryan A. Plummer, Justin Salamon, Oriol Nieto, Bryan Russell, Kate Saenko, Language-Guided Audio-Visual Source Separation via Trimodal Consistency, CVPR 2023, [arxiv] [code]

Tl; dr: We can use language as a bridge to teach models to perform audio-visual sound separation.

2021

Ajay Divakaran, Karan Sikka, Arijit Ray, Xiao Lin, Yi Yao, User-targeted content generation using multimodal embeddings, US Patent App. 17/191,698 [webpage]

Kamran Alipour, Arijit Ray, Xiao Lin, Michael Cogswell, Jurgen Schulze, Yi Yao, Giedrius Burachas, Improving Users' Mental Model with Attention-directed Counterfactual Edits, 2021 Applied AI Letters (Wiley) [pdf]

Arijit Ray, Michael Cogswell, Xiao Lin, Kamran Alipour, Ajay Divakaran, Yi Yao, Giedrius Burachas, Knowing What VQA Does Not: Pointing to Error-Inducing Regions to Improve Explanation Helpfulness, 2021 Applied AI Letters (Wiley), [pdf] [arXiv] [Project Page]

2019

Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lee, Giedrius Burachas, Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation , (EMNLP 2019), also at CVPR-W 2019 VQA and Visual Dialog Workshop, [arXiv], [bibTex] [Data]

Arijit Ray, Yi Yao, Rakesh Kumar, Ajay Divakaran, Giedrius Burachas, Can You Explain That: Lucid Explanations Help Human-AI Collaboratve Image Retrieval , (AAAI-HCOMP 2019), [arXiv], [bibTex] [press coverage]

2016

Arijit Ray, Gordon Christie, Mohit Bansal, Dhruv Batra, Devi Parikh, "Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions.", (EMNLP 2016). [pdf] [code] [Video]

Course/Mini Projects:

Prashant Chandrasekar, Xuan Zhang, Saurabh Chakravarty, Arijit Ray, John Krulick, and Alla Rozovskaya, "The Virginia Tech System at CoNLL-2016 Shared Task on Shallow Discourse Parsing", CoNLL Shared Task (2016).

The Art of Deep Connection - Towards Natural and Pragmatic Conversational Agent Interactions. [Master's Thesis], Virginia Tech E-Library, 2017

Make RBF Networks Fast Again- Exploiting Multi-Threaded Computing to Speed Up RBF Networks, Multiprocessor Programming Class Project, Fall 2016, [draft paper] [code]

Object Prediction using Image Context: Predict next object in an image reasoned on present image context in a sequential manner, Computer Vision Class Project Fall 2015

Online Demo for Predicting Plausibility of Common Sense Assertions: Enter a three-phrase tuple to assess the plausibility score based on a joint language-vision common-sense reasoning, Class Project, Fall 2015

Learning to Listen: Matching Cover songs with Original Productions: Match Original Songs to Cover Songs using an Ensemble of Supervised and Unsupervised Approaches, Machine Learning Class Project, Fall 2015.

Ray, Arijit, Kishan Prudhvi Guddanti, and N. Chellammal. "An Approach to Intelligent Traction Control Using Regression Networks and Anomaly Detection.", Junior (3rd Year) Semester Project, Fall 2013, published in Springer Applied Artificial Intelligence 29.6 (2015): 597-616.

Tutorials

Press Coverage

Collaborators

Some of the amazing people I have been fortunate to work with:

Prof. Dhruv Batra (at Virgina Tech), Prof. Stefan Lee (at Virgina Tech and SRI Intl.), Dr. Dhruv Mahajan (at FAIR), Dr. Filip Radenovic (at FAIR), Dr. Abhimanyu Dubey (at FAIR), Dr. Ajay Divakaran (at SRI Intl), Dr. Yi Yao (at SRI Intl), Dr. Giedrius Burachas (at SRI Intl), Dr. Kezhen Chen (at Google X, Mineral)

Hobbies

When I am not training LLMs, I love going to techno (a subgenre of electronic music) fests, making latte art, and engineering simple gadgets. In middle school, I opened an informal research society to encourage fellow students to take an interest in science by constructing simple gadgets. We won multiple accolades in school and city-level exhibitions.

Miscellanea

Contact Me

Have a question?

Best way to reach me would be to drop an email to array at bu dot edu.