How to scrape all links from a website

Tue, 02 May 2023 00:00:00 +0000

Simply copy-paste the following one-liner to Chrome Dev Tools. To limit the results, set the reg exp placeholder to contain some sensible value, or replace the function call with return value true.

Array.prototype.slice.call( document.getElementsByTagName('a'), 0 )
  .map(i => i.href)
  .filter(i => i.match('reg exp placeholder'))
  .forEach(i => console.log(i))

Here is an explanation for what it does:

Selects all the elements on the document.
Converts the HTMLCollection object returned by getElementsByTagName into an Array by using the Array.prototype.slice method.
The slice() method returns a shallow copy of a portion of an array into a new array object selected from begin to end (end not included) where begin and end represent the index of items in that array. Here, it selects all the items starting from index 0 to the end of the array by passing 0 as the first argument to slice() method.
Maps the href attribute of each of the selected elements into an array using the Array.prototype.map method. The resulting array will contain a list of all the href attributes of the elements.
Filters the array obtained in the previous step by selecting only those elements that match a regular expression pattern. The pattern is a placeholder and should be replaced with an actual regular expression to match the desired elements. This is achieved using the Array.prototype.filter method.
Loops through the resulting filtered array using the forEach() method and logs each element to the console using console.log() method. This will output the list of href attributes of the elements that match the given regular expression pattern to the console.

Dev Tools on Arto's dev time by-products

How to scrape all links from a website