Written by Manas Gaur, assistant professor in computer science and electrical engineering, UMBC

ChatGPT and other AI chatbots based on large language models are known to occasionally make things up, including scientific and legal citations. It turns out that measuring how accurate an AI model’s citations are is a good way of assessing the model’s reasoning abilities.

An AI model “reasons” by breaking down a query into steps and working through them in order. Think of how you learned to solve math word problems in school.

Ideally, to generate citations an AI model would understand the key concepts in a document, generate a ranked list of relevant papers to cite, and provide convincing reasoning for how each suggested paper supports the corresponding text. It would highlight specific connections between the text and the cited research, clarifying why each source matters.

The question is, can today’s models be trusted to make these connections and provide clear reasoning that justifies their source choices? The answer goes beyond citation accuracy to address how useful and accurate large language models are for any information retrieval purpose.

I’m a computer scientist. My colleagues − researchers from the AI Institute at the University of South Carolina, Ohio State University and University of Maryland Baltimore County − and I have developed the Reasons benchmark to test how well large language models can automatically generate research citations and provide understandable reasoning.

We used the benchmark to compare the performance of two popular AI reasoning models, DeepSeek’s R1 and OpenAI’s o1. Though DeepSeek made headlines with its stunning efficiency and cost-effectiveness, the Chinese upstart has a way to go to match OpenAI’s reasoning performance.

Sentence specific

The accuracy of citations has a lot to do with whether the AI model is reasoning about information at the sentence level rather than paragraph or document level. Paragraph-level and document-level citations can be thought of as throwing a large chunk of information into a large language model and asking it to provide many citations.

In this process, the large language model overgeneralizes and misinterprets individual sentences. The user ends up with citations that explain the whole paragraph or document, not the relatively fine-grained information in the sentence.

Further, reasoning suffers when you ask the large language model to read through an entire document. These models mostly rely on memorizing patterns that they typically are better at finding at the beginning and end of longer texts than in the middle. This makes it difficult for them to fully understand all the important information throughout a long document.

Large language models get confused because paragraphs and documents hold a lot of information, which affects citation generation and the reasoning process. Consequently, reasoning from large language models over paragraphs and documents becomes more like summarizing or paraphrasing.

The Reasons benchmark addresses this weakness by examining large language models’ citation generation and reasoning.

Testing citations and reasoning

Following the release of DeepSeek R1 in January 2025, we wanted to examine its accuracy in generating citations and its quality of reasoning and compare it with OpenAI’s o1 model. We created a paragraph that had sentences from different sources, gave the models individual sentences from this paragraph, and asked for citations and reasoning.

To start our test, we developed a small test bed of about 4,100 research articles around four key topics that are related to human brains and computer science: neurons and cognition, human-computer interaction, databases and artificial intelligence. We evaluated the models using two measures: F-1 score, which measures how accurate the provided citation is, and hallucination rate, which measures how sound the model’s reasoning is − that is, how often it produces an inaccurate or misleading response.

Our testing revealed significant performance differences between OpenAI o1 and DeepSeek R1 across different scientific domains. OpenAI’s o1 did well connecting information between different subjects, such as understanding how research on neurons and cognition connects to human-computer interaction and then to concepts in artificial intelligence, while remaining accurate. Its performance metrics consistently outpaced DeepSeek R1’s across all evaluation categories, especially in reducing hallucinations and successfully completing assigned tasks.

OpenAI o1 was better at combining ideas semantically, whereas R1 focused on making sure it generated a response for every attribution task, which in turn increased hallucination during reasoning. OpenAI o1 had a hallucination rate of approximately 35% compared with DeepSeek R1’s rate of nearly 85% in the attribution-based reasoning task.

In terms of accuracy and linguistic competence, OpenAI o1 scored about 0.65 on the F-1 test, which means it was right about 65% of the time when answering questions. It also scored about 0.70 on the BLEU test, which measures how well a language model writes in natural language. These are pretty good scores.

DeepSeek R1 scored lower, with about 0.35 on the F-1 test, meaning it was right about 35% of the time. However, its BLEU score was only about 0.2, which means its writing wasn’t as natural-sounding as OpenAI’s o1. This shows that o1 was better at presenting that information in clear, natural language.

OpenAI holds the advantage

On other benchmarks, DeepSeek R1 performs on par with OpenAI o1 on math, coding and scientific reasoning tasks. But the substantial difference on our benchmark suggests that o1 provides more reliable information, while R1 struggles with factual consistency.

Though we included other models in our comprehensive testing, the performance gap between o1 and R1 specifically highlights the current competitive landscape in AI development, with OpenAI’s offering maintaining a significant advantage in reasoning and knowledge integration capabilities.

These results suggest that OpenAI still has a leg up when it comes to source attribution and reasoning, possibly due to the nature and volume of the data it was trained on. The company recently announced its deep research tool, which can create reports with citations, ask follow-up questions and provide reasoning for the generated response.

The jury is still out on the tool’s value for researchers, but the caveat remains for everyone: Double-check all citations an AI gives you.

This article is republished from The Conversation under a Creative Commons license. Read the original article and see more than 250 UMBC articles available in The Conversation.

When associate professor of visual arts Kelley Bell thinks back to her childhood growing up in the Capitol Hill area of Washington, D.C., she recalls fond memories of playing in her neighborhood playground, locally dubbed as “Turtle Park.”

“I remembered the concrete structures behind the turtle in the playground and it really inspired me to want to figure out how they came to be,” Bell says. 

The public park, officially named Marion Park, was given the “Turtle Park” moniker because of the large concrete turtle structure that lived in the center of its playground, designed by Virginia Dortch Dorazio in 1953. Bell, M.F.A. ’06, imaging and digital arts, remembers Turtle Park as a prominent staple of her adolescence and decades later, the park also acts as the inspiration behind her latest project “Fantastic Village,” which will be on display at the Baltimore Museum of Art (BMA) from April 27 through July 27 as part of the museum’s Baker Artist Awards exhibition. 

The power of design

Bell was a 2024 recipient of the Greater Baltimore Cultural Alliance’s Baker Artist Award. The BMA exhibit will feature 20 works from five recent Baker Artist Award recipients, including Bell’s “Fantastic Village.” The colorful geometries of Bell’s installation draw on memories of now-demolished playgrounds of her youth in Washington, D.C., and Baltimore’s iconic rowhomes, bridging together the two places she calls home. 

Bell creates vibrant projection-mapping works on a grand scale and gallery installations that emphasize joy, playfulness, community, and human connection. Before pursuing her current career as a visual artist, designer, and educator, Bell initially thought she’d be a lawyer. She believed that since her father was a speechwriter for President Gerald Ford and her mother was a teacher, she’d continue the family’s lineage of public service work. 

However, “drawing in class was always a problem with me to the degree that it’s written on a lot of my report cards, saying, ‘Kelley has a lot of attention issues.’ I never thought of art as a career,” says Bell.

Growing up in the nation’s capital, Bell was heavily influenced by the city’s independent music scene. During her wanderings through downtown Washington as a teenager, a chance encounter with a repurposed “five and dime” store (then home to the Washington Project for the Arts) changed everything.

“When I walked in that store, I never saw anything like it,” Bell recalls, “There were zines, posters, paintings, and tapes for bands all around. It was a window into this idea that art doesn’t have to be this picture on the wall and that was an amazing idea to me even at a young age.”

After graduating from Pratt, Bell landed a designer role for a record pressing plant that worked with dance music labels. Continuing her work in the entertainment industry, she freelanced for MTV Networks for several years before she was encouraged to explore Baltimore after a recommendation from a fellow designer who boasted about the city’s arts scene. A one-day exploration of the city convinced Bell that Baltimore was the place for her.

Bell was accepted into the Pratt Institute in Brooklyn, New York where she initially majored in illustration. But once there, she was introduced to the concept and the power of design. “A friend explained that designers construct the existing world, and I believe that’s an amazing power to have. I quickly switched my major after that,” says Bell.

In 1997, Bell moved into Baltimore’s now-defunct H&H Arts Building, a five-story warehouse where local artists lived and worked for more than two decades. It was at H&H where Bell was initially connected toperformer and fellow UMBC alum Beatrix “Trixie Little” Burneston ’99, visual and performing arts, a founding member of Fluid Movement, a Baltimore-based organization that utilizes public spaces to create joy and community through performance art that most notably includes quirky water ballet performances in Baltimore City public pools.

“Fluid Movement did a rendition of the opera ‘Carmen’ with hotdogs on sticks. They designed a stage and costumes for hotdogs and had a composer to do the score,” shares Bell. “To see people pour that much love and effort into something that ultimately is incredibly silly and unusual is, to me, a heroic act.”

Bell worked with Fluid Movement as a designer and the group then became a long-time client that she worked with throughout the years. Bell has also collaborated with a number of organizations and art collectives across Baltimore, including working with the Enoch Pratt Library and the community art space Creative Alliance. Bell’s project “The Clock Strikes 100,” a series of short animations designed for and projected within the Baltimore’s historic Bromo Seltzer Art Tower, was designated “Best Public Artwork” in Baltimore Magazine’s “Best of Baltimore 2011” issue.

“Projections, Inflatables, and Artistic Spectacles”

Bell’s first introduction to UMBC was on a snowy day in 2001. Her friends recommended UMBC’s campus as a great place for sledding due to the campus’ hilly landscape. Soon after, Bell decided to expand her skillset and enrolled in UMBC’s M.F.A program. 

“I was like a kid in a candy store with all of the stuff that I could do. It was a new world being opened up to me,” says Bell.

After experiencing difficulties in executing her thesis project to her satisfaction, Bell was advised by her then-thesis chair Preminda Jacob to consider extending her time in the program for an additional year. 

“That extra year was the year that I figured out projection mapping. Doing guerilla projection projects changed the way that I thought about making art. That has been a form of art that’s served me since I’ve graduated from UMBC and it was all because of that fourth year,” says Bell. Guerilla projection mapping—the use of projectors to display images, animations, or videos onto various surfaces in public spaces without formal authorization or permits—helped to inform the foundation of Bell’s artistic pursuits.  

Bell’s large-scale projection-mapping works and public installations have been featured in national and international festivals for almost two decades. Among her notable works include “The Herd,” a 2018 installation that was part of Light City Festival in Baltimore City. The project featured more than 300 solar powered inflatables that were placed into Baltimore’s Inner Harbor as a call to action for healthy waterways in the harbor. 

In addition to her artistic practice, Bell has been an educator at UMBC since 2008. Earlier this year, Bell presented a retrospective of her work and her research practice with the presentation “Projections, Inflatables, and Artistic Spectacles,” organized by the UMBC’s Center for Innovation, Research, and Creativity in the Arts (CIRCA). 

“The CIRCA talk was a breakthrough. It was the first time that I took in the full scope of starting with an M.F.A. and how that work led up to the BMA exhibit,” she says. 

The motivation behind Bell’s BMA exhibit “Fantastic Village” largely stemmed from her collaboration with UMBC theater professor Collete Searls on the “Enchanted Jangle” installation that was part of the 2024 Sweaty Eyeballs: Animation Adjacent gallery exhibition, curated by Corrie Parks, associate professor of visual arts. The installation is described as an “epic cardboard fort your five-year-old self dreamed of.”

“There’s a playfulness about it but also a deeply-weird and creepy aspect to it. This is the place I like to dwell—giving people something new and something that they can’t figure out is something that is very important to me in my work.”

Temping rooms available for 1 or 2 females from end of May till 23rd August. Details are as follows:

Fully furnished townhouse.
Room rent : 636 for 1 female and 318 each if shared by 2 females.
Temping duration: May end till Aug end .
2 mins walk to the paradise bus stop.
10 mins walk to the campus .
Mattresses already available in the room.
For details contact : +14437036282.

OSP Coffee Chat Series

The Office of Sponsored Programs Training Team would like to invite you to join us for a casual yet informative, virtual coffee chat. Bringe your coffee and your questions to join us in this short information session. This invitation is for you if you are a faculty, staff, or otherwise active participant in sponsored research at UMBC and would like to learn more from your OSP Team on various topics including: What is OSP? What is F&A? What is Cost Share? And more!

In an effort to create an even more open path to communication and education, and real-time answers, OSP’s virtual coffee chat series kicks off May 8^th, 11:30 am -12:00 pm.

Present Topic: OSP 101 – What is the Office of Sponsored Programs All About?

Presenters: Jamie Cole, Chelsea Raff, and Allison Martins Machado

Description: The present topic will be all about OSP including their duties and function within UMBC.

At the end of each session there will be time for some Q&A. Please feel free to comment with questions ahead of time to be answered during the current or future sessions. If you have questions that pertain to a specific grant, please reach out to your Grants Manager individually.

To Join Via Webex on May 8th, please use the information below:

https://umbc.webex.com/umbc/j.php?MTID=m4f8fbaedd21081b07d06223f1d702881

Meeting number (access code): 2870 366 4002
Meeting password: a7wWmEpKh32

TAP TO JOIN FROM A MOBILE DEVICE (ATTENDEES ONLY)
+1-202-860-2110,,28703664002## tel:%2B1-202-860-2110,,*01*28703664002%23%23*01* United States Toll (Washington D.C.)

JOIN BY PHONE
+1-202-860-2110 United States Toll (Washington D.C.)

Global call-in numbers
https://umbc.webex.com/umbc/globalcallin.php?MTID=m8a0c379e2d7238fcd5b5c7797d896c02

JOIN FROM A VIDEO SYSTEM OR APPLICATION
Dial sip:28703664002@umbc.webex.com
You can also dial 173.243.2.68 and enter your meeting number.

Can't join the meeting?
https://collaborationhelp.cisco.com/article/WBX000029055

Thank you!

OSP training Team