Where do I find text to analyze?
The Internet is full of content that anyone can access, but just becuase you can see it doesn't mean you can download the content and turn it into a dataset. This kind of activity is considered "reuse." It is important to always check and understand the copyright status and license (when applicable) for any restrictions on reuse of any text you want to mine. Librarians are great resources to consult when looking for resources to use for text mining.
Government Documents
- Data.govThe U.S. government's open data portal. Limit by file type to locate textual data resources. Files are accessible as XML files, PDFs, HTML files, or TXT files.
- GovInfo (formerly FDsys)The Government Publishing Office's publication platform for digital copies of documents across all three branches of government.
- CIA FOIA Reading Room (declassified)Declassified FOIA Requests posted on the CIA web site. Contains curated historical collections.
- Europa Open Data Portal"The EU Open Data Portal is your single point of access to a growing range of data produced by the institutions and other bodies of the European Union. Data are free to use, reuse, link and redistribute for commercial or non-commercial purposes." From the portal landing page.
- FRED APIThe FRED® API is a web service that allows developers to write programs and build applications that retrieve economic data from the FRED® and ALFRED® websites hosted by the Economic Research Division of the Federal Reserve Bank of St. Louis. Requests can be customized according to data source, release, category, series, and other preferences.
- HathiTrust Digital Library This link opens in a new window
This link opens in a new window
This link opens in a new window
(Searches through citations and the full text)
HathiTrust is a partnership of major research institutions and libraries working to ensure that the cultural record is preserved and digitally accessible. A major source of digitized books and other documents. Some of the collection if available for free to all, and additional content is available to Carleton faculty, staff, and students.
Primary Source Collections
- HathiTrust Digital Library This link opens in a new window
- This link opens in a new window
- This link opens in a new window
(Searches through citations and the full text)
HathiTrust is a partnership of major research institutions and libraries working to ensure that the cultural record is preserved and digitally accessible. A major source of digitized books and other documents. Some of the collection if available for free to all, and additional content is available to Carleton faculty, staff, and students. - DPLA: Digital Public Library of America This link opens in a new window
- This link opens in a new window
DPLA is a U.S. project to create a large-scale public digital library. Contains public domain and openly licensed content held by the United States’ archives, libraries, museums, and other cultural heritage institutions. Always check the rights information before downloading.
- Internet archive
- This link opens in a new window
The Internet Archive is a non-profit that was founded to build an Internet library. Not everything is licensed for reuse. Look carefully before using.
- Europeana This link opens in a new window
- This link opens in a new window
Europeana is a multi-lingual online collection of millions of digitized items from European museums, libraries, archives, and multi-media collections. Link goes to public domain texts and those licensed for reuse.
- QualiBankFrom the UK Data Archive, this collection of qualitative surveys, interviews, and open ended questions. Start by browsing the interviews by using the Resource Type filter.
- Project GutenbergA massive collection of out-of-copyright books, transcribed and downloadable in multiple formats. Free for anyone to use.
- All Carleton Digital CollectionsA local digital collection of Carleton publications and images from Carleton's Visual Resources Collection, the Carleton Archives, and teaching collections compiled by Carleton faculty.
- Google Books Advanced SearchFind books available for download by limiting to "Full View Only."
Database APIs & CONTENTdm
DO NOT SCRAPE LIBRARY DATABASES
Why? You're stealing protected information, and that violation of Terms of Service can get access turned off for the entire campus.
But, there are options!
Always ask your library liaison if you don't know how to access the underlying data in a library database -- we can help you do it legally! Depending on the database vendor, we may be able to get hard drives of all the underlying data, or we can help to set up API access for you.
- Research Consultations with Librarians by SubjectClick on a librarian's name to find contact and schedule information.
- About CONTENTdm APIsInstructions on how to use the CONTENTdm API.
- Carleton CONTENTdm Digital CollectionsThere are many digital collections hosted in CONTENTdm, including Course Catalogs, Northfield historical material, and other digital objects used in classes.
- GDELTEnormous dataset of media from around the world, coverage from 1979-forward.
