Java web crawler

This webcrawler is a java application that takes in a URL and returns a list of websites that can be found from that URL at a specific depth of the given URL. This was the first project that I had worked together with others to design the concept and idea of the project.

While creating this webcrawler I got a chance to expand my knowledge of computer science. From this project I learned how to extract links from HTML documents, use regular expressions to filter through links, and implement graph traversal algorithms. This project also allowed me to use many data structures that I had learned in ICS 211 (Data Structues), such as Hashtables and LinkedLists. This project also taught me to develop skills of looking thing’s up on StackOverflow to solve the many problems that I faced during the project. One of the challenges I faced during this project was figuring out how to design a project this large and taught me to how break a large problem down into smaller parts. This project exposed me to both Depth First Search and Breadth First Search and showed me one of many valuable usages for these algorithms.

Source: devgav/WebCrawler