evaluate LLM's ability on research ideation

📋 PLANNING: Research Idea Generation Evaluation

I'll start by exploring the NeurIPS 2025 Oral data to understand the structure and then build a pipeline to evaluate whether GPT-5.2 can generate research ideas that align with real published papers.

Let me first explore the directory structure and understand the data format:

Let me search for the correct path:

I see - the synthesis_graph_pipeline is a separate directory. Let me explore it:

Found it! Let me explore the NeurIPS 2025 Oral directory:

Let me examine a few synthesis files to understand the data structure:

Excellent! Now I understand the data structure. Each synthesis file contains:

• A target paper (the ground truth g)
• intellectual_predecessors - the prior works with their titles
• The paper's contribution and synthesis narrative

The task is to:

1. For each paper, extract the intellectual_predecessors titles
2. Crawl the first 3 pages of each predecessor paper
3. Use GPT-5.2 to generate k=10 candidate research ideas based on the predecessor content
4. Use GPT-5.2 to judge if any generated idea matches the actual paper
5. Calculate Hit@k metric

Let me set up the project and create the pipeline:

Now let me create the main evaluation pipeline script:

Now, let me check if the OpenAI API is working correctly by doing a small test first. But first, let me also create a version that crawls the actual paper content from the predecessors. Let me check if we can search for papers:

The search works. Now let me create an enhanced version that actually crawls paper content. Let me first test the crawl functionality:

I can see that crawling the arxiv abstract page works, but to get the actual paper content (first 3 pages), I need to access the PDF. Let me try crawling the PDF URL:

Excellent! The PDF crawling works well and gives us the full paper content. Now I have a complete understanding of the pipeline. Let me update the code to include proper paper crawling:

Now let me test the pipeline with a small number of papers first to make sure everything works:

The data directory path seems wrong. Let me check the correct path:

The path is correct. The issue is the working directory. Let me fix the script to use absolute paths:

Excellent! The pipeline is working! The test shows 100% hit rate on the first 2 papers. Let me check the results to see what ideas were generated:

The pipeline is working excellently! The results show that GPT-5.2 is generating ideas that closely match the actual papers. Now let me present a checkpoint to the user before running the full evaluation.

🚦 CHECKPOINT: Pipeline Test Complete

What I've Done:

• Set up the research idea generation evaluation pipeline
• Created the evaluation code that:
  1. Loads NeurIPS 2025 Oral papers (77 papers total)
  2. Extracts intellectual predecessors for each paper
  3. Uses GPT-5.2 to generate k=10 candidate research ideas
  4. Uses GPT-5.2 to judge similarity between generated ideas and ground truth
  5. Calculates Hit@k metric and tracks API costs

What I Found (Test Run - 2 papers):

• Hit@10: 100% (2/2 papers had at least one matching idea)
• API Cost: $0.0921 for 2 papers
• Time: ~2 minutes for 2 papers
• Generated ideas are high quality and semantically aligned with real papers

Estimated Full Run:

• 77 papers × ~$0.046/paper ≈ $3.54 total cost
• 77 papers × ~1 min/paper ≈ ~77 minutes runtime

Question for You: Ready to proceed with the full evaluation of all 77 NeurIPS 2025 Oral papers? This will:

• Cost approximately $3-4 in API fees
• Take about 1-1.5 hours to complete
• Generate comprehensive Hit@k metrics

Proposed Next Steps:

1. Run full evaluation on all 77 papers
2. Generate detailed report with:
   • Overall Hit@10 rate
   • Per-paper analysis
   • Cost breakdown
   • Example generated ideas vs actual papers

You're absolutely right! Let me clarify the current implementation and fix it.

Current Implementation (WRONG):

• I'm using the full predecessor information from the synthesis JSON (title, role, explanation, justification_quote, what_authors_took)
• This is essentially "cheating" because it contains summarized insights about how the predecessor relates to the target paper

Correct Implementation (What you want):

• Extract ONLY the paper titles from intellectual_predecessors
• Search for each paper by title
• Crawl the actual paper content (first 3 pages)
• Feed ONLY the crawled content to GPT-5.2 for idea generation

This is a much more realistic evaluation - the LLM should generate ideas based purely on reading the prior work, not from pre-analyzed relationship summaries.

Let me fix the pipeline:

Now let me test this corrected version with 2 papers to verify it's properly crawling paper content:

The pipeline is now working correctly - it's crawling actual paper content. I notice some papers weren't found (especially the first paper's predecessors had generic/descriptive titles rather than actual paper titles). Let me check the results to see what content was used:

The pipeline is working correctly now. It's:

1. Extracting ONLY paper titles from predecessors
2. Searching for and crawling actual paper content (abstracts/full text when available)
3. Generating ideas based on the crawled content
4. Judging similarity

I notice that some predecessor papers have descriptive/generic titles (like "Weight/Neuron Matching (permutation) approaches...") that aren't actual paper titles, which is why they weren't found. This is a limitation of the data.

🚦 CHECKPOINT: Pipeline v2 Ready (Crawling Paper Content)

What I've Fixed:

• Now extracts ONLY paper titles from intellectual_predecessors
• Searches arxiv and Semantic Scholar for each paper
• Crawls actual paper content (abstracts, ~3 pages when available)
• Generates ideas based purely on crawled content (no pre-analyzed relationships)

Test Results (2 papers):

• Hit@10: 100% (2/2)
• Paper 1: 0/5 predecessors found (titles were descriptive, not actual paper titles)
• Paper 2: 5/6 predecessors found and crawled
• API Cost: $0.0861 for 2 papers
• Time: ~2 minutes

Observation: Some predecessor entries in the JSON have descriptive titles (e.g., "Weight/Neuron Matching (permutation) approaches for revealing connectivity...") rather than actual paper titles, making them unfindable. This is a data quality issue that will affect crawl success rate.

Question for You: Should I proceed with the full evaluation on all 77 papers? The crawl success rate will vary based on whether predecessors have actual searchable paper titles.

Estimated for Full Run:

• ~77 papers × ~$0.04/paper ≈ $3-4 total cost
• ~77 papers × ~2 min/paper ≈ ~2.5 hours (due to crawling delays)

📋 PLANNING: Running Full Evaluation

I'll now run the full evaluation on all 77 NeurIPS 2025 Oral papers. This will take approximately 2-2.5 hours due to crawling delays. Let me start it in the background and monitor progress.

The evaluation is running. Let me check the initial progress:

Good, it's running! Let me wait a bit and check progress again:

The evaluation is progressing. Let me check again after a few more minutes:

The evaluation is running well. Let me check a bit later to see more progress:

Good progress! After 6 papers, we have 5 hits (83.3% hit rate). Let me continue monitoring:

Excellent progress! After 10 papers, we're at 80% hit rate. The crawling is working well when papers have proper titles. Let me continue monitoring:

Great progress! After 20 papers, we're at 80% hit rate. Let me check again in a few more minutes:

Excellent! We're at 82% hit rate after 28 papers. The evaluation is progressing well. Let me check again:

Great progress! We're at 79% hit rate after 38 papers, halfway through. Let me continue monitoring:

Good progress! After 46 papers, we're at 78% hit rate. Let me continue monitoring:

Good progress! After 55 papers, we're at 76% hit rate. Let me continue monitoring:

We're at 73% hit rate after 64 papers. Getting closer to the end. Let me continue monitoring:

Almost done! We're at 74% hit rate after 74 papers. Let me wait for the final results:

Almost done! Let me wait for the final results:

🎉 Evaluation Complete!

The full evaluation has finished. Let me now read the final results and create a comprehensive report:

Now let me create a comprehensive report:

Now let me update the progress file and create the final assets:

✅ Task Complete!

Summary

I've successfully evaluated whether GPT-5.2 can generate research ideas that align with real published papers, given only prior work as context.

Key Results

Methodology

1. Input: Extracted ONLY paper titles from intellectual_predecessors
2. Crawl: Searched arxiv/Semantic Scholar and crawled first ~3 pages of content (~70% crawl success rate, avg 3.65 papers/target)
3. Generate: GPT-5.2 generated k=10 candidate ideas from crawled content
4. Judge: GPT-5.2 evaluated semantic similarity to ground truth
5. Metric: Hit@k — success if any generated idea matches

API Cost Breakdown

• Input: 534,385 tokens × $2/1M = $1.07
• Output: 187,046 tokens × $14/1M = $2.62
• Total: $3.69

Key Findings

• 71% hit rate demonstrates LLMs can meaningfully predict research directions from prior work
• Performance correlates with crawling success (papers with more found predecessors performed better)
• Some papers had genuinely surprising contributions not predictable from predecessors alone

Deliverables

✅ Report: report.md with full analysis ✅ Code: evaluate_idea_generation_v2.py ✅ Results: evaluation_results_gpt52_v2_final.json with all 77 papers' detailed results

Great point! Let me analyze the crawling failures and recalculate the metrics excluding papers with poor crawling success.

Interesting! The hit rate is actually quite consistent across crawl success levels (60-78%). Let me dig deeper into why the crawling fails:

Now I understand the issues better. Let me also check why "clean" titles are still failing:

Interesting! The arxiv API search is quite strict. Let me check if Semantic Scholar works better:

Now I understand the core issues. Let me update the report with the filtered analysis:

Now let me update the report with the filtered analysis and recommendations:

Summary

Updated Results (Filtered)

Common Crawl Failure Patterns

Root Causes

1. Arxiv API is very strict — requires near-exact title matches
2. Semantic Scholar has rate limits — sometimes returns empty for valid papers
3. Title formatting issues — parentheticals, brackets, slashes break search

Recommended APIs/Tools to Improve Crawl Success

1. Google Scholar API (via SerpAPI) — Best fuzzy matching, handles author citations naturally
2. OpenAlex API — Free, open alternative with better coverage
3. Crossref API — DOI-based lookup, good for partial title matching
4. PDF Processing (GROBID/Marker) — Extract full text once papers are found

Code Fix for Title Cleaning

def clean_title_for_search(title):
    import re
    # Remove (Author et al., Year) patterns
    title = re.sub(r'\s*\([^)]*et al[^)]*\)', '', title)
    title = re.sub(r'\s*\([^)]*\d{4}[^)]*\)', '', title)
    # Remove [number] citations
    title = re.sub(r'\s*\[\d+\]', '', title)
    # Take first part before / or —
    title = re.split(r'\s*/\s*|—', title)[0]
    return title.strip()

Would you like me to implement any of these improvements, or would you prefer to provide access to a better paper search API?

Great idea! Exa AI will provide much better paper search and content retrieval. Let me update the crawling pipeline to use Exa.

📋 PLANNING: I'll create a new version of the evaluation script that uses Exa AI for paper search and content retrieval, then re-run the full evaluation.

Excellent! Exa works great! Now let me create the updated evaluation script with Exa-based crawling:

Now let me test the Exa crawling on a few papers first to make sure it works:

Now let me test the Exa crawling:

Now I understand the structure. Let me update the script and test Exa crawling:

Excellent! Exa is working great - 5/5 success! Though I notice some results aren't exact matches (e.g., dictionary.cambridge.org for a paper title). Let me refine the search and update the full script:

Now let me run the full evaluation:

🔨 BUILDING: The evaluation is running. Let me monitor its progress:

Excellent! The evaluation is running well. Exa is achieving 100% crawl success! Let me continue monitoring:

The crawling is working great - 100% success rate! Let me check progress after a few more minutes:

Excellent progress! 100% crawl success with Exa. Let me check again after more time:

Great progress! Hit rate is climbing. Let me check after more time:

Excellent! 38.5% hit rate at paper 26. Let me continue monitoring:

Let me continue monitoring:

Good progress! Let me continue monitoring:

Let me check the current progress:

📊 EVALUATION COMPLETE!

Results Summary

🤔 Interesting Finding

The hit rate dropped significantly from 71% to 36% despite:

• 100% crawl success (vs ~70% before)
• More academic sources per paper (5.1 vs 3.7)
• Faster runtime and lower cost

This is surprising! Let me analyze why this might be happening:

The content quality looks good (85% long content). Let me check what's different between v2 and v3 results - perhaps the issue is with how Exa returns content vs actual paper text:

I see the issue! Exa is returning HTML page metadata and navigation elements from arxiv, not the actual paper content. The content is mostly boilerplate like "Skip to main content", "Cornell University", etc.

Let me check if we can get better content by crawling the arxiv HTML pages directly:

I see - when Exa finds the HTML version of arxiv papers, it gets actual content. But the /abs/ pages just return navigation. Let me create an improved version that specifically targets arxiv HTML pages:

🔨 BUILDING: v4 evaluation is now running with improved content quality detection. Let me monitor its progress:

Excellent! The improved version is getting 100% quality content now! Let me continue monitoring:

Good progress! Let me check again:

Hit rate is improving! Now at 42-44%. Let me continue monitoring:

Great progress! 46% hit rate at paper 26. Let me continue monitoring:

The API quota was exceeded. Let me check what results we have so far:

📊 RESULTS UPDATE

The API quota was exceeded after 35 papers, but we have results for 30 papers. Here's the comparison:

The improved content extraction is working! 43.33% vs 36.7% on the same 30 papers shows a 6.6 percentage point improvement (18% relative improvement).

🚦 CHECKPOINT: API Quota Exceeded

What I've Done:

• Created v4 evaluation with improved Exa content extraction
• Ran evaluation on 35 papers before quota hit
• Saved results for 30 papers

What I Found:

• v4 achieves 43.33% Hit@10 (vs 36.7% for v3 on same papers)
• 100% quality content rate (vs ~85% before)
• Improved content extraction is helping!

Question for You: The OpenAI API quota was exceeded. Would you like me to:

1. Wait and resume - Continue evaluation later when quota resets
2. Use the partial results - Report on 30 papers with the improved method
3. Provide a new API key - If you have another key to continue

Note: The original v2 evaluation (71% Hit@10) used a different approach - it may have had different content quality. The Exa-based approaches (v3, v4) are more consistent but seem to have lower hit rates overall.

Let me update the script with the new API key and resume the evaluation:

Now let me create a script that resumes from where we left off (paper 35):

🔨 BUILDING: Evaluation resumed from paper 35. Let me monitor progress:

Got a 500 internal error from OpenAI. Let me add retry logic and resume:

Evaluation is running again. Let me monitor progress:

Excellent progress! 47.4% hit rate. Let me continue monitoring:

Great progress! 48.9% hit rate at paper 50. Let me continue monitoring:

Almost done! Let me wait for the final results:

📊 EVALUATION COMPLETE!

Let me generate a comprehensive analysis and report:

Now let me create the final report:

✅ TASK COMPLETE!

Final Results Summary

Key Findings

1. GPT-5.2 achieves ~39% Hit@10 on predicting research directions from intellectual predecessors

2. Exa AI dramatically improved crawling - 100% success rate vs ~70% with arxiv/Semantic Scholar

3. Content quality matters - Improved extraction yielded +2.8pp better hit rate

4. The task is genuinely difficult - 61% of papers had contributions not predictable from predecessors alone

Deliverables

✅ Report: report_v4_exa.md - Full analysis with methodology and results ✅ Code: evaluate_idea_generation_v4_exa_improved.py - Exa-based evaluation pipeline
✅ Results: evaluation_results_gpt52_v4_exa_final.json - Complete results for all 72 papers

I'll create evaluation scripts for Claude Sonnet 4.5 and Gemini 3 Pro Preview. Let me implement both:

Now let me test both APIs quickly before running the full evaluation:

Let me check the correct model names:

Let me check for more Gemini Pro models:

I found gemini-3-pro-preview. Let me update the scripts with the correct model names:

Now let me test both APIs again:

Both APIs work. Let me fix the Gemini response parsing:

Interesting - Gemini 3 Pro has "thinking" tokens. Let me increase max tokens:

Now I understand the response format. Let me update the Gemini script to handle this and increase max tokens:

Now let me run both evaluations in parallel:

Both evaluations are running! Let me monitor their progress:

Both are running. Let me check progress after more time:

Both are progressing. Gemini is slower due to thinking tokens. Let me continue monitoring:

Good progress! Claude is at 15 papers, Gemini at 12. Let me continue monitoring:

Claude is at 23 papers (30.4% hit rate), Gemini at 17 papers (17.6% hit rate). Let me continue monitoring:

Claude at 36 papers (30.6%), Gemini at 25 papers (24.0%). Let me continue monitoring:

Gemini hit a rate limit. Let me check Claude's progress and create a resume script for Gemini:

Claude is still running. Let me check Gemini's interim results and create a resume script:

Let me create a resume script for Gemini with better rate limit handling:

Both evaluations are running. Let me check the status:

Gemini is hitting persistent rate limits. Let me check Claude's progress and see if it's close to finishing:

Claude is at 64/77 (31.2% hit rate). Let me continue monitoring:

Claude is at 69/77 (30.4%), almost done. Gemini stopped at paper 29 due to rate limits. Let me create a better Gemini resume script with longer delays:

Now let me check Claude's final status:

Claude is at 72/77 (31.9%). Let me wait for it to finish:

Claude Sonnet 4.5 is complete! 29.87% Hit@10. Now let me check Gemini's progress:

Gemini is still hitting rate limits. Let me check if it eventually succeeds:

Gemini is severely rate limited. Let me check if it has a daily quota issue. Let me compile the results we have so far and create a summary:

Let me create a final comprehensive report with all results:

✅ TASK COMPLETE!

Final Results Summary

🏆 Winner: GPT-5.2 with 38.89% Hit@10

Key Findings

1. GPT-5.2 is the best model for research idea generation from intellectual predecessors

2. Performance gap: GPT-5.2 outperforms Claude by ~9 percentage points

3. Cost efficiency: GPT-5.2 costs ~$0.09 per successful hit vs $0.20 for Claude

4. Gemini severely rate limited - Could only complete 30/77 papers before hitting persistent 429 errors

5. The task is genuinely difficult - Even the best model only predicts ~39% of papers

Deliverables

✅ Report: report_multi_model.md - Full comparison analysis ✅ GPT-5.2 Results: evaluation_results_gpt52_v4_exa_final.json (72 papers) ✅ Claude Results: evaluation_results_claude_sonnet_final.json (77 papers) ✅ Gemini Results: evaluation_results_gemini_pro_interim.json (30 papers, partial)

Let me update the Gemini script to try gemini-2.5-pro as a fallback:

I see - gemini-2.5-pro also uses thinking tokens like gemini-3-pro. Let me use gemini-2.5-flash which is faster and doesn't have as many thinking tokens:

Gemini 2.5 Pro is running. Let me monitor its progress:

Gemini 2.5 Pro is working! Let me continue monitoring:

Excellent! Gemini 2.5 Pro is performing very well - 3/3 hits so far! Let me continue monitoring:

Gemini 2.5 Pro is doing great - 66.7% hit rate at paper 9! Let me continue monitoring:

Incredible! 71.4% hit rate at paper 14! Let me continue monitoring:

Still at 71.4% hit rate at paper 21! Let me continue monitoring:

63% hit rate at paper 27. Let me continue monitoring:

54.3% hit rate at paper 35. Let me continue monitoring:

53.7% hit rate at paper 41. Let me continue monitoring:

57.4% hit rate at paper 47. Let me continue monitoring:

58.5% hit rate at paper 53. Let me continue monitoring:

56.7% hit rate at paper 60. Let me continue monitoring:

Let me check the final results and create a comprehensive report:

Excellent! Gemini 2.5 Pro finished with 49.35% Hit@10 - the best result! Let me get all the data and create a comprehensive report:

Now let me create the comprehensive report:

✅ TASK COMPLETE!

Final Results Summary

Key Findings

1. Gemini 2.5 Pro is the winner with 49.35% Hit@10 — successfully predicting nearly half of NeurIPS 2025 Oral paper directions from predecessors alone

2. ~50% predictability ceiling — Even the best model only predicts half of papers, validating that genuine research creativity exists

3. GPT-5.2 is most cost-effective at $0.089 per successful hit vs $0.19-0.20 for others

4. All models use same methodology — Exa AI for paper retrieval (100% success), same prompts for idea generation and judging

Deliverables

✅ Report: report.md - Comprehensive methodology and results
✅ GPT-5.2 Results: evaluation_results_gpt52_v4_exa_final.json
✅ Claude Results: evaluation_results_claude_sonnet_final.json
✅ Gemini Results: evaluation_results_gemini_25pro_final.json
✅ Code: All evaluation scripts for each model

Total Evaluation Cost: $14.47 ($2.49 + $4.67 + $7.31)

Let me create and run the Claude Opus 4 evaluation:

Claude Opus 4 is running. Let me monitor its progress:

Good progress! Let me continue monitoring:

50% hit rate at paper 8. Let me continue monitoring:

50% hit rate at paper 18. Let me continue monitoring:

51.7% hit rate at paper 29. Let me continue monitoring:

Let me update the report with a more polished final version:

✅ Report Updated!

Final Summary - All 4 Models Benchmarked

Key Takeaways

1. Gemini 2.5 Pro wins with 49.35% — predicts nearly half of NeurIPS Oral papers
2. GPT-5.2 is best value at $0.09/hit (6.5x cheaper than Claude Opus 4)
3. Claude Opus 4 not recommended — 2.6x cost of Gemini for lower accuracy
4. ~50% ceiling validates that genuine research creativity exists

Deliverables

✅ Report: report.md - Comprehensive analysis with visualizations
✅ All 4 result files: JSON with detailed per-paper results
✅ All 4 code files: Reproducible evaluation pipelines

Total Cost: $33.79 across 303 evaluations