Aggregating Aggregation: Finding Relevant Jobs

The job hunting process is like climbing a mountain. Each time the mountain crests, you assume it is the top only to be disappointed when there is another ascent. C'est la vie. To that end, I'm trying to implement some robotic legs to do my grunt work to the next ascent.

This is a function that takes the argument that specify a search to Indeed.com, read the page, find the job links, and scrapes the third party redirect postings for relevant keywords. The non-function implementation works well but this bugs out quickly.

require(scrapeR) Keywords <- c("quantitative","physics","entry level","recent","college graduate", " r ","python","forecast","forecasting","analyst","analytics","analysis", "model","modeling","predictive","prediction","analyzed","predicted", "forecasted","new graduate","recent graduate", "data analyst","data") htmlCleaner <- function(htmlString){ return(gsub("<.*?>","",htmlString))} jobFinder <- function(searchWords,area,radius,pageToSearch){ for (l in seq(1,pageToSearch)){ searchDescrip <- gsub(" ","+",searchWords) area <- gsub(" ","+",area) indeedPage <- paste("http://www.indeed.com/jobs?q=",searchDescrip, "&l=",area,"&radius=",as.character(radius),"&sort=date", "&start=",as.character(l*10),sep="") #finds the HTML page, reads it in, and finds the third party links #by indeeds html format. Eliminates blank space. Calls lines that #refer to a job posting. print(indeedPage) html <- tolower(readLines(indeedPage))#searches the html lines for urls. cleanHtml <- html[html!=""] lines <- grep("jobposting",cleanHtml) for (i in seq(1,length(lines))){ #Creates a chunk of html that has hyperlink. Finds hyperlink. chunkHTML <- cleanHtml[lines[i]:(lines[i]+8)] linkLine <- grep("href=\\\"(.*?)\\\"",chunkHTML) test <- gregexpr("href=\\\"(.*?)\\\"", chunkHTML[linkLine]) hLink <- regmatches(chunkHTML[linkLine],test)[[1]] hLink <- paste("www.indeed.com",substr(hLink,7,nchar(hLink)-1),sep="") jPage <- scrape(hLink,follow=TRUE,parse=TRUE) jPage <- htmlCleaner(as(jPage[[1]],"character")) wordMatches <- vector(mode="numeric",length=length(Keywords)) names(wordMatches) <- Keywords for (j in seq(1,length(Keywords))) { if (gregexpr(Keywords[j],jPage)[[1]][1]>0) {wordMatches[j] <- length(gregexpr(Keywords[j],jPage)[[1]])} else{wordMatches[j] <- 0} } print(wordMatches) print(hLink) } } }


Example inputs as "entry level analyst"==search terms, "sacramento ca"==geographic area, 100==radius, 1==the number of pages to search. The bug appears on the fifth third party scrape, with values==defs. Debugging the function takes me to the line that scrapes the HTML but I'm still not sure where the error comes up. Obviously a work in progress, but if I can automate looking through 60 pages of Indeed.com a day, I can save myself a nice chunk of time.

> jobFinder("entry level analyst","sacramento ca",100,1) [1] "http://www.indeed.com/jobs?q=entry+level+analyst&l=sacramento+ca&radius=100&sort=date&start=10" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 0 forecasting analyst analytics analysis 0 0 0 0 model modeling predictive prediction 0 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 0 [1] "www.indeed.com/rc/clk?jk=0dc735391ec9e5fd" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 0 forecasting analyst analytics analysis 0 1 2 0 model modeling predictive prediction 3 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 33 [1] "www.indeed.com/rc/clk?jk=6dd34d0431d25416" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 0 forecasting analyst analytics analysis 0 0 0 4 model modeling predictive prediction 2 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 3 [1] "www.indeed.com/rc/clk?jk=f186fe17c3346164" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 2 forecasting analyst analytics analysis 0 0 0 0 model modeling predictive prediction 0 0 0 0 analyzed predicted forecasted new graduate 0 0 1 0 recent graduate data analyst data 0 0 7 [1] "www.indeed.com/rc/clk?jk=cacce23a3c446ed7" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 2 0 0 forecasting analyst analytics analysis 0 4 2 3 model modeling predictive prediction 1 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 15 [1] "www.indeed.com/rc/clk?jk=8382c8ba0875d4f8" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 0 forecasting analyst analytics analysis 0 0 0 0 model modeling predictive prediction 0 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 0 [1] "www.indeed.com/rc/clk?jk=b56639458dcd2619" Error in value == defs : 'value' is missing