Aggregating Aggregation: "Exception Handling or: How I learned to Stop Worrying and Let R Handle Imperfections"

The code for parsing Indeed.com by keyword and area in the previous post has some primitive features. It looks for exact patterns, and based on the usual syntax of Indeed HTML, subtracts a fixed number of characters to make a usable link which I can re-paste.

I think the error I have experienced comes from this fixed number subtraction taking away/leaving a relevant/irrelevant character. It works for most links, but not all. Ideally, I would just skip the exceptions and let the loop continue through.

R does this through tryCatch(), which has a confusing documentation. Reading through this posting from the Working With Data blog helped me in writing a function that loads as many pages of Indeed as specified.

require(scrapeR) Keywords <- c("quantitative","physics","entry level","recent","college graduate", " r ","python","forecast","forecasting","analyst","analytics","analysis", "model","modeling","predictive","prediction","analyzed","predicted", "forecasted","new graduate","recent graduate", "data analyst","data") htmlCleaner <- function(htmlString){ return(gsub("<.*?>","",htmlString))} jobFinder <- function(searchWords,area,radius,pageToSearch){ counter <- 0 for (l in seq(1,pageToSearch)){ searchDescrip <- gsub(" ","+",searchWords) area <- gsub(" ","+",area) indeedPage <- paste("http://www.indeed.com/jobs?q=",searchDescrip, "&l=",area,"&radius=",as.character(radius),"&sort=date", "&start=",as.character(l*10),sep="") #finds the HTML page, reads it in, and finds the third party links #by indeeds html format. Eliminates blank space. Calls lines that #refer to a job posting. print(indeedPage) html <- tolower(readLines(indeedPage))#searches the html lines for urls. cleanHtml <- html[html!=""] lines <- grep("jobposting",cleanHtml) for (i in seq(1,length(lines))){ tryCatch({ #Creates a chunk of html that has hyperlink. Finds hyperlink. chunkHTML <- cleanHtml[lines[i]:(lines[i]+8)] linkLine <- grep("href=\\\"(.*?)\\\"",chunkHTML) test <- gregexpr("href=\\\"(.*?)\\\"", chunkHTML[linkLine]) hLink <- regmatches(chunkHTML[linkLine],test)[[1]] hLink <- paste("www.indeed.com",substr(hLink,7,nchar(hLink)-1),sep="") jPage <- scrape(hLink,follow=TRUE,parse=TRUE) jPage <- htmlCleaner(as(jPage[[1]],"character")) wordMatches <- vector(mode="numeric",length=length(Keywords)) names(wordMatches) <- Keywords for (j in seq(1,length(Keywords))) { if (gregexpr(Keywords[j],jPage)[[1]][1]>0) {wordMatches[j] <- length(gregexpr(Keywords[j],jPage)[[1]])} else{wordMatches[j] <- 0} } print(wordMatches) print(hLink) counter <- counter+1 print(counter) },warning=function(war){ print("problem") },error=function(err){ print("error") },finally={ }) } } }


tryCatch takes your code and runs it. If an error or warning occurs, it redirects to the warning/error "handler", below the tryCatch{}. In redirecting, it doesn't stop the main loop with an error so my problem seems avoided although I don't understand what happens entirely. I ran through 5 pages of Indeed postings for analyst, and it spit out 50 instances of the expected output (as shown in the previous post).

Now I need to figure out what kind of output I want this to give, how I can speed up the execution, and automate a list of user inputs for both the keywords searched and multiple search terms, ie, "analyst","physics",... rather than the single "analyst".

but the legs are operational!