WARC file for Department of Atmospheric and Environmental Sciences - College of Arts and Sciences - University at Albany - SUNY, 2017 January 17

Acquisition information:

crawl: 261028

Crawl Rules

Ignore Robots.txt for www.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for www.alumni.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for www.ualbanysports.com (last updated 2016-02-11)

Ignore Robots.txt for library.albany.edu (last updated 2017-05-19)

Ignore Robots.txt for alumni.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for asrc.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for atmos.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for bioinformatics.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for cela.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for choose.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for cs.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for csda.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for imls.ctg.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for www.ctg.albany.edu (last updated 2017-05-19)

Ignore Robots.txt for cwig.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for events.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for hr.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for ibl.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for illiad.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for liblogs.albany.edu (last updated 2017-05-19)

Ignore Robots.txt for libguides.library.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for scholarsarchive.library.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for listserv.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for m.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for math.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for omega.math.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for mumford.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for nyjm.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for pdp.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for resnet.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for rit.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for cyberphysics.rit.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for rna.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for slsc.albany.edu (last updated 2017-05-19)

Ignore Robots.txt for uaems.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for uapps.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for wiki.albany.edu (last updated 2016-02-11)

Block host dev.library.albany.edu

Crawl Times

start_date: 2017-01-17T17:03:42Z

original_start_date: 2017-01-17T17:03:42Z

last_resumption: None

processing_end_date: 2017-01-23T00:48:05Z

end_date: 2017-01-22T19:31:33Z

elapsed_ms: 440845477

Crawl Types

type: MONTHLY

recurrence_type: MONTHLY

pdfs_only: False

test: False

Crawl Limits

time_limit: 432000

document_limit: None

byte_limit: None

crawl_stop_requested: None

Crawl Results

status: FINISHED_TIME_LIMIT

discovered_count: 6856243

novel_count: 810250

duplicate_count: 844972

resumption_count: 0

queued_count: 5201021

downloaded_count: 1655222

download_failures: 351

warc_revisit_count: 844920

warc_url_count: 1655090

total_data_in_kbs: 260833049

duplicate_bytes: 210669517964

warc_compressed_bytes: 15598206209

Crawl Technical Details

doc_rate: 3.75

kb_rate: 591.0

Physical / technical requirements:
Researchers interested in data analysis with web archives may request a WARC file. WARC files are very large and difficult to work with. Your request may take time to process, and we may be unable to deliver your request remotely. Please consult an archivist if you are interested in advanced research with web archives.

Using these materials

Access:
The archives are open to the public and anyone is welcome to visit and view the collections.
Collection restrictions:
Access to this collection is restricted because it is unprocessed. Portions of the collection may contain recent administrative records and/or personally identifiable information. While it is likely that portions of the collection may be viewed, access must be managed by an archivist.
Collection terms of access:
The researcher assumes full responsibility for conforming with the laws of copyright. Whenever possible, the M.E. Grenander Department of Special Collections and Archives will provide information about copyright owners and other restrictions, but the legal determination ultimately rests with the researcher. Requests for permission to publish material from this collection should be discussed with the Head of Special Collections and Archives.

About the Archives

Access options

Ask an Archivist

Ask a question or schedule an individualized meeting to discuss archival materials and potential research needs.

Ask a Question Book a Meet

Schedule a Visit

Archival materials can be viewed in-person in our reading room. We recommend making an appointment to ensure materials are available when you arrive.

Schedule a Visit Hours

Make a Remote Request

We may also be able to deliver digital scans remotely for a fee.

Request Fees