WARC file for University at Albany - SUNY - Home Page, 2017 May 22

Acquisition information:

crawl: 302323

Crawl Rules

Ignore Robots.txt for www.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for www.alumni.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for www.ualbanysports.com (last updated 2016-02-11)

Ignore Robots.txt for library.albany.edu (last updated 2017-05-19)

Ignore Robots.txt for alumni.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for asrc.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for atmos.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for bioinformatics.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for cela.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for choose.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for cs.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for csda.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for imls.ctg.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for www.ctg.albany.edu (last updated 2017-05-19)

Ignore Robots.txt for cwig.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for events.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for hr.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for ibl.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for illiad.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for liblogs.albany.edu (last updated 2017-05-19)

Ignore Robots.txt for libguides.library.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for scholarsarchive.library.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for listserv.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for m.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for math.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for omega.math.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for mumford.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for nyjm.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for pdp.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for resnet.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for rit.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for cyberphysics.rit.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for rna.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for slsc.albany.edu (last updated 2017-05-19)

Ignore Robots.txt for uaems.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for uapps.albany.edu (last updated 2016-02-11)

Ignore Robots.txt for wiki.albany.edu (last updated 2016-02-11)

Block host dev.library.albany.edu

Host Rule Type C CONTAINS meg.library.albany.edu:8080/archive/search? (last updated 2017-05-11)

Limit host pbs.twimg.com to 10 documents

Limit host s-media-cache-ak0.pinimg.com to 10 documents

Limit host pbs.twimg.com to 10 documents

Limit host s-media-cache-ak0.pinimg.com to 10 documents

Limit host abs.twimg.com to 10 documents

Limit host www.youtube.com to 10 documents

Limit host www.pinterest.com to 10 documents

Limit host s.pinimg.com to 10 documents

Limit host mobile.twitter.com to 10 documents

Limit host twitter.com to 10 documents

Limit host mobile.twitter.com to 10 documents

Limit host twitter.com to 10 documents

Limit host plus.google.com to 10 documents

Limit host twitter.com to 10 documents

Limit host plus.google.com to 10 documents

Limit host yt3.ggpht.com to 10 documents

Limit host plus.google.com to 10 documents

Limit host yt3.ggpht.com to 10 documents

Limit host r19---sn-n4v7kn7e.googlevideo.com to 10 documents

Limit host es.pinterest.com to 10 documents

Limit host www.pinterest.pt to 10 documents

Limit host kr.pinterest.com to 10 documents

Limit host www.pinterest.se to 10 documents

Limit host sk.pinterest.com to 10 documents

Limit host www.pinterest.se to 10 documents

Limit host sk.pinterest.com to 10 documents

Limit host dk.pinterest.com to 10 documents

Limit host sk.pinterest.com to 10 documents

Limit host ar.pinterest.com to 10 documents

Limit host dk.pinterest.com to 10 documents

Limit host fi.pinterest.com to 10 documents

Limit host ar.pinterest.com to 10 documents

Limit host chrome.google.com to 10 documents

Limit host fi.pinterest.com to 10 documents

Limit host gaming.youtube.com to 10 documents

Limit host chrome.google.com to 10 documents

Limit host cl.pinterest.com to 10 documents

Limit host gaming.youtube.com to 10 documents

Limit host it.pinterest.com to 10 documents

Limit host cl.pinterest.com to 10 documents

Limit host au.pinterest.com to 10 documents

Limit host it.pinterest.com to 10 documents

Limit host tr.pinterest.com to 10 documents

Limit host co.pinterest.com to 10 documents

Limit host au.pinterest.com to 10 documents

Limit host co.pinterest.com to 10 documents

Limit host tr.pinterest.com to 10 documents

Limit host ro.pinterest.com to 10 documents

Limit host co.pinterest.com to 10 documents

Limit host za.pinterest.com to 10 documents

Limit host ro.pinterest.com to 10 documents

Limit host id.pinterest.com to 10 documents

Limit host za.pinterest.com to 10 documents

Limit host gr.pinterest.com to 10 documents

Limit host www.pinterest.de to 10 documents

Limit host id.pinterest.com to 10 documents

Limit host www.pinterest.de to 10 documents

Limit host cz.pinterest.com to 10 documents

Limit host gr.pinterest.com to 10 documents

Limit host cz.pinterest.com to 10 documents

Limit host hu.pinterest.com to 10 documents

Limit host www.pinterest.de to 10 documents

Limit host hu.pinterest.com to 10 documents

Limit host payments.google.com to 10 documents

Limit host cz.pinterest.com to 10 documents

Limit host payments.google.com to 10 documents

Limit host nz.pinterest.com to 10 documents

Limit host hu.pinterest.com to 10 documents

Limit host nz.pinterest.com to 10 documents

Limit host schema.org to 10 documents

Limit host payments.google.com to 10 documents

Limit host in.pinterest.com to 10 documents

Limit host nz.pinterest.com to 10 documents

Limit host schema.org to 10 documents

Limit host fonts.gstatic.com to 10 documents

Limit host schema.org to 10 documents

Limit host in.pinterest.com to 10 documents

Limit host jp.pinterest.com to 10 documents

Limit host in.pinterest.com to 10 documents

Limit host fonts.gstatic.com to 10 documents

Limit host pinterest.com to 10 documents

Limit host fonts.gstatic.com to 10 documents

Limit host jp.pinterest.com to 10 documents

Limit host s.ytimg.com to 10 documents

Limit host jp.pinterest.com to 10 documents

Limit host pinterest.com to 10 documents

Limit host uk.pinterest.com to 10 documents

Limit host pinterest.com to 10 documents

Limit host s.ytimg.com to 10 documents

Limit host ru.pinterest.com to 10 documents

Limit host s.ytimg.com to 10 documents

Limit host uk.pinterest.com to 10 documents

Limit host ru.pinterest.com to 10 documents

Limit host br.pinterest.com to 10 documents

Limit host uk.pinterest.com to 10 documents

Limit host no.pinterest.com to 10 documents

Limit host br.pinterest.com to 10 documents

Limit host ru.pinterest.com to 10 documents

Limit host accounts.google.com to 10 documents

Limit host no.pinterest.com to 10 documents

Limit host br.pinterest.com to 10 documents

Limit host www.gstatic.com to 10 documents

Limit host accounts.google.com to 10 documents

Limit host no.pinterest.com to 10 documents

Limit host nl.pinterest.com to 10 documents

Limit host www.gstatic.com to 10 documents

Limit host accounts.google.com to 10 documents

Limit host fr.pinterest.com to 10 documents

Limit host nl.pinterest.com to 10 documents

Limit host www.gstatic.com to 10 documents

Limit host pl.pinterest.com to 10 documents

Limit host fr.pinterest.com to 10 documents

Limit host nl.pinterest.com to 10 documents

Limit host get.adobe.com to 10 documents

Limit host pl.pinterest.com to 10 documents

Limit host fr.pinterest.com to 10 documents

Limit host get.adobe.com to 10 documents

Limit host dns: to 10 documents

Limit host pl.pinterest.com to 10 documents

Limit host dns: to 10 documents

Limit host connect.facebook.net to 10 documents

Limit host get.adobe.com to 10 documents

Limit host connect.facebook.net to 10 documents

Limit host vine.co to 10 documents

Limit host dns: to 10 documents

Limit host vine.co to 10 documents

Limit host input.name to 10 documents

Limit host connect.facebook.net to 10 documents

Limit host input.name to 10 documents

Limit host www.addthis.com to 10 documents

Limit host vine.co to 10 documents

Limit host www.addthis.com to 10 documents

Limit host m.youtube.com to 10 documents

Limit host input.name to 10 documents

Limit host m.youtube.com to 10 documents

Limit host i9.ytimg.com to 10 documents

Limit host www.addthis.com to 10 documents

Limit host i9.ytimg.com to 10 documents

Limit host www.nyraen.org to 10 documents

Limit host m.youtube.com to 10 documents

Limit host www.nyraen.org to 10 documents

Limit host meinvz.net to 10 documents

Limit host i9.ytimg.com to 10 documents

Limit host meinvz.net to 10 documents

Limit host ualbany.alpinesystemsinc.com to 10 documents

Limit host www.nyraen.org to 10 documents

Limit host ualbany.alpinesystemsinc.com to 10 documents

Limit host www.xing.com to 10 documents

Limit host meinvz.net to 10 documents

Limit host www.xing.com to 10 documents

Limit host staticxx.facebook.com to 10 documents

Limit host mar.gar.in to 10 documents

Limit host www.xing.com to 10 documents

Limit host mar.gar.in to 10 documents

Limit host ct.pinterest.com to 10 documents

Limit host staticxx.facebook.com to 10 documents

Limit host ct.pinterest.com to 10 documents

Limit host www.adobe.com to 10 documents

Limit host mar.gar.in to 10 documents

Limit host www.adobe.com to 10 documents

Limit host es.pn to 10 documents

Limit host ct.pinterest.com to 10 documents

Limit host es.pn to 10 documents

Limit host citeulike.org to 10 documents

Limit host www.adobe.com to 10 documents

Limit host citeulike.org to 10 documents

Limit host fai.informazione.it to 10 documents

Limit host es.pn to 10 documents

Limit host fai.informazione.it to 10 documents

Limit host ads-latest.pinterest.com to 10 documents

Limit host citeulike.org to 10 documents

Limit host ads-latest.pinterest.com to 10 documents

Limit host cbks1.googleapis.com to 10 documents

Limit host fai.informazione.it to 10 documents

Limit host cbks1.googleapis.com to 10 documents

Limit host drive.google.com to 10 documents

Limit host ads-latest.pinterest.com to 10 documents

Limit host archive.org to 10 documents

Limit host cbks1.googleapis.com to 10 documents

Limit host cdn.youvisit.com to 10 documents

Limit host drive.google.com to 10 documents

Limit host taringa.net to 10 documents

Limit host archive.org to 10 documents

Limit host vybrali.sme.sk to 10 documents

Limit host cdn.youvisit.com to 10 documents

Limit host com.google.android.youtube to 10 documents

Limit host taringa.net to 10 documents

Limit host about.me to 10 documents

Limit host vybrali.sme.sk to 10 documents

Limit host uqshyy.gi to 10 documents

Limit host com.google.android.youtube to 10 documents

Limit host nmnathletics.com to 10 documents

Limit host about.me to 10 documents

Limit host steamcommunity.com to 10 documents

Limit host uqshyy.gi to 10 documents

Limit host getpocket.com to 10 documents

Limit host nmnathletics.com to 10 documents

Limit host www.insidelacrosse.com to 10 documents

Limit host www.informz.net to 10 documents

Limit host steamcommunity.com to 10 documents

Limit host www.informz.net to 10 documents

Limit host www.w3.org to 10 documents

Limit host getpocket.com to 10 documents

Limit host www.w3.org to 10 documents

Limit host qr-adv.com to 10 documents

Limit host www.insidelacrosse.com to 10 documents

Limit host qr-adv.com to 10 documents

Limit host apps.albany.edu to 10 documents

Limit host www.informz.net to 10 documents

Limit host apps.albany.edu to 10 documents

Limit host meetvibe.com to 10 documents

Limit host www.w3.org to 10 documents

Limit host meetvibe.com to 10 documents

Limit host smb.cdnak.dal.neulion.com to 10 documents

Limit host qr-adv.com to 10 documents

Limit host smb.cdnak.dal.neulion.com to 10 documents

Limit host apply.commonapp.org to 10 documents

Limit host apps.albany.edu to 10 documents

Limit host apply.commonapp.org to 10 documents

Limit host www.atmos.albany.edu to 10 documents

Limit host meetvibe.com to 10 documents

Limit host www.atmos.albany.edu to 10 documents

Limit host code.google.com to 10 documents

Limit host smb.cdnak.dal.neulion.com to 10 documents

Limit host code.google.com to 10 documents

Limit host www.pinterest.info to 10 documents

Limit host apply.commonapp.org to 10 documents

Limit host www.pinterest.info to 10 documents

Limit host www.ualbanydining.com to 10 documents

Limit host mts1.googleapis.com to 10 documents

Limit host code.google.com to 10 documents

Limit host mts1.googleapis.com to 10 documents

Limit host www.pinterest.info to 10 documents

Limit host mts1.googleapis.com to 10 documents

Crawl Times

start_date: 2017-05-22T15:27:32Z

original_start_date: 2017-05-22T15:27:32Z

last_resumption: None

processing_end_date: 2017-05-22T15:56:34Z

end_date: 2017-05-22T15:38:26Z

elapsed_ms: 502536

Crawl Types

type: DAILY

recurrence_type: DAILY

pdfs_only: False

test: False

Crawl Limits

time_limit: 82800

document_limit: 1000

byte_limit: None

crawl_stop_requested: None

Crawl Results

status: FINISHED_DOCUMENT_LIMIT

discovered_count: 2694

novel_count: 248

duplicate_count: 754

resumption_count: 0

queued_count: 1692

downloaded_count: 1002

download_failures: 0

warc_revisit_count: 733

warc_url_count: 981

total_data_in_kbs: 38590

duplicate_bytes: 27177594

warc_compressed_bytes: 4309429

Crawl Technical Details

doc_rate: 1.99

kb_rate: 76.0

Physical / technical requirements:

Researchers interested in data analysis with web archives may request a WARC file. WARC files are very large and difficult to work with. Your request may take time to process, and we may be unable to deliver your request remotely. Please consult an archivist if you are interested in advanced research with web archives.

Using these materials

Access:: The archives are open to the public and anyone is welcome to visit and view the collections.

Collection restrictions:: Access to this collection is unrestricted.
Collection terms of access:: The University Archives are eager to hear from any copyright owners who are not properly identified so that appropriate information may be provided in the future.