Repository navigation
[QUESTION]: Handling of multi-word keywords in Scopus query construction #4
Description
Your Question
While reviewing the metadata collection workflow, I noticed that multi-word keywords appear to be inserted directly into the Scopus query URL. I have not yet verified whether this causes incorrect behavior in the Scopus API because I currently do not have access to a Scopus API key.
Is this the expected behavior, or should multi-word keywords be URL encoded or quoted before being passed to Scopus?
What I've Already Tried
- Checked the documentation: Yes
- Searched existing issues: Yes
- Installed ComProScanner locally and traced the metadata collection workflow.
- Followed the query path:
main_fetch() → _fetch_paginated_data() → fetch_and_process_data() → _construct_url()
- Generated a test URL using the keyword:
thermal conductivity
Generated URL:
https://api.elsevier.com/content/search/scopus?query=PUBYEAR+%3D+2024+thermal conductivity&count=200&cursor=*
I noticed that the space between "thermal" and "conductivity" is preserved in the generated URL and I could not find explicit URL encoding in this code path.
I currently do not have access to a Scopus API key, so I have not yet verified whether this causes incorrect behavior or whether Scopus handles it internally.
Context
I'm trying to understand whether this could be related to the multi-word keyword issue previously discussed in Slack.
My goal is to help validate and improve the metadata collection workflow before testing with Scopus access.
System & Environment
- OS: Windows 10
- Processor: Intel Core i5 (adjust if needed)
- Installed RAM: 8 GB (adjust if needed)
- GPU Enabled: No
- Terminal: Windows Command Prompt (CMD)
- Python environment: Python 3.12.10
Additional context
I can provide additional testing results once I obtain access to a Scopus API key.
Activity
aritraroy24 commentedon Jun 16, 2026
Hi @WilmerGaspar, thanks for the detailed report! At first glance this looks like a bug, though it won't completely break the workflow. Here's what may be affected:
Metadata Collection
%20by the Scopus backend. While this prevents the workflow from breaking, it reduces search accuracy — e.g., passingthermal conductivitymay also return results forelectrical conductivityandthermal expansion, which costs unnecessary API calls.Article Processing
thermal conductivity_metadata.csv. Most OSes handle this fine, but replacing spaces with_is better practice for scripting.is_sql_db=True(False by default), a space in the table name could cause issues. That said, SQLAlchemy wraps space-containing names in quotes automatically, so it should still work.Data Extraction, Cleaning, Evaluation, and Visualisation
✅ Recommended fix
_in the main keyword before the metadata collection step._before any regex matching in article processing and the data extraction flow.If you'd like to contribute a fix, I'm happy to assign this to you!
P.S.: For getting the Scopus API, please refer to the API key guide (provided your institution has a subscription to the Scopus database)
WilmerGaspar commentedon Jun 17, 2026
Thanks a lot for checking this, Aritra. I’m glad the report was useful.
Yes, I’d be happy to work on a small fix for this issue. I understand that the workflow may not completely break, but improving the handling of multi-word keywords could help with query accuracy and file/table naming consistency.
I currently don’t have institutional Scopus API access, so I’ll focus only on the local keyword preprocessing and query-construction behavior, without making claims about live API validation.
I can start by following your recommended fix: replacing spaces with
_before metadata collection and normalizing or stripping_before regex matching in the later processing steps.If this sounds good, please assign the issue to me and I’ll start reviewing the relevant code path.
aritraroy24 commentedon Jun 17, 2026
Hi @WilmerGaspar, thanks for showing interest in working on the fix. I'll assign this task to you. Please refer to the CONTRIBUTING.md before submitting the pull request.
Regarding the Scopus API key, I ran some access tests with an API key generated from a personal Gmail-based Elsevier developer account. The results are as follows:
You will have full access (
cursor=*compatibility for multiple-page crawl) if you are on an institutional IP which has the Elsevier subscription.If you are not on institutional IP, you will still get some valid results. But you can't use
cursor=*in the request URL (¶meter should also be removed as the next header parameter is being removed). So, the request URLs will be (properly space-encoded):https://api.elsevier.com/content/search/scopus?query=PUBYEAR+%3D+2024+thermal%20AND%20conductivityhttps://api.elsevier.com/content/search/scopus?query=PUBYEAR+%3D+2024+thermal%20AND%20conductivity&apiKey=your-scopus-api-keyMerge pull request #5 from WilmerGaspar/fix/multi-word-keyword-handling
aritraroy24 commentedon Jun 24, 2026
Fixed: #5 by @WilmerGaspar
fix: multi-word keyword handling
WilmerGaspar commentedon Jun 24, 2026
Thank you, Aritra. I really appreciate your guidance and review throughout the process. I’m glad I could contribute to ComProScanner and help fix this issue.
It was a great learning experience for me, and I’d be happy to help with future issues if there’s another suitable task.