metascraper

A library to easily extract unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more.

• What is it
• Getting started
• What data it detects
• The cloud API solution
• How it works
• Importing rules
• Rules bundles
  • Official
  • Community
• API
  • constructor(rules)
    • rules
  • metascraper(options)
    • options
    • html
    • htmlDom
    • omitPropNames
    • pickPropNames
    • rules
    • url
    • validateUrl
• Environment Variables
  • METASCRAPER_RE2
• Benchmark
• License

---

What is it

The metascraper library allows you to easily scrape metadata from an article on the web using Open Graph metadata, regular HTML metadata, and a series of fallbacks.

It follows a few principles:

• Ensure a high accuracy for online articles by default.
• Make it simple to add new rules or override existing ones.
• Don't restrict rules to CSS selectors or text accessors.

Getting started

Below is a real example of extracting metadata from a live website. The same logic shown here is running online and can be tested directly at microlink.io/meta:

metascraper requires two inputs: The target URL and the HTML markup behind that URL.

There are multiple ways to retrieve the HTML markup, but it needs to be as accurate as possible.

For that reason, we developed html-get, which uses a headless browser to retrieve HTML in a way that works seamlessly with metascraper.

const getHTML = require('html-get')

/**
 * `browserless` will be passed to `html-get`
 * as driver for getting the rendered HTML.
 */
const browserless = require('browserless')()

const getContent = async url => {
  // create a browser context inside the main Chromium process
  const browserContext = browserless.createContext()
  const promise = getHTML(url, { getBrowserless: () => browserContext })
  // close browser resources before return the result
  promise.then(() => browserContext).then(browser => browser.destroyContext())
  return promise
}

/**
 * `metascraper` is a collection of tiny packages,
 * so you can just use what you actually need.
 */
const metascraper = require('metascraper')([
  require('metascraper-author')(),
  require('metascraper-date')(),
  require('metascraper-description')(),
  require('metascraper-image')(),
  require('metascraper-logo')(),
  require('metascraper-publisher')(),
  require('metascraper-title')(),
  require('metascraper-url')()
])

/**
 * The main logic
 */
getContent('https://microlink.io')
  .then(metascraper)
  .then(metadata => console.log(metadata))
  .then(browserless.close)
  .then(process.exit)

The output will be something like:

{
  "author": "Microlink HQ",
  "date": "2022-07-10T22:53:04.856Z",
  "description": "Enter a URL, receive information. Normalize metadata. Get HTML markup. Take a screenshot. Identify tech stack. Generate a PDF. Automate web scraping. Run Lighthouse",
  "image": "https://cdn.microlink.io/logo/banner.jpeg",
  "logo": "https://cdn.microlink.io/logo/logo.png",
  "publisher": "Microlink",
  "title": "Turns websites into data — Microlink",
  "url": "https://microlink.io/"
}

What data it detects

Note: Custom metadata detection can be defined using a rule bundle.

Here is an example of the metadata that metascraper can detect:

• audio — e.g. https://cf-media.sndcdn.com/U78RIfDPV6ok.128.mp3 A audio URL that best represents the article.

• author — e.g. Noah Kulwin A human-readable representation of the author's name.

• date — e.g. 2016-05-27T00:00:00.000Z An ISO 8601 representation of the date the article was published.

• description — e.g. Venture capitalists are raising money at the fastest rate... The publisher's chosen description of the article.

• video — e.g. https://assets.entrepreneur.com/content/preview.mp4 A video URL that best represents the article.

• image — e.g. https://assets.entrepreneur.com/content/3x2/1300/20160504155601-GettyImages-174457162.jpeg An image URL that best represents the article.

• lang — e.g. en An ISO 639-1 representation of the url content language.

• logo — e.g. https://entrepreneur.com/favicon180x180.png An image URL that best represents the publisher brand.

• publisher — e.g. Fast Company A human-readable representation of the publisher's name.

• title — e.g. Meet Wall Street's New A.I. Sheriffs The publisher's chosen title of the article.

• url — e.g. http://motherboard.vice.com/read/google-wins-trial-against-oracle-saves-9-billion The URL of the article.

The cloud API solution

Running this at scale means operating headless browsers, proxies, and antibot workarounds.

If you don’t want to manage that infrastructure, you can use the fully managed Microlink API.

It automatically handles proxy rotation, paywalls, bot detection, and restricted platforms such as major social networks, while scaling on demand.

Pricing is pay-as-you-go and starts for free.

How it works

metascraper is built out of rules bundles.

It is designed to be extensible. You can compose your own transformation pipeline using existing rules or create your own.

Rule bundles are collections of HTML selectors targeting a specific property. When you load the library, it implicitly loads the core rules.

Each set of rules loads a set of selectors to extract a specific value.

Rules are ordered by priority. The first rule to successfully resolve the value stops the process. The order goes from most specific to most generic.

Rules work as fallbacks for one another:

• If the first rule fails, then it falls back on the second rule.
• If the second rule fails, it is time for the third rule.
• Etc.

metascraper does this until it finishes all the rules or finds the first rule that resolves the value.

Importing rules

metascraper exports a constructor that need to be initialized providing a collection of rules to load:

const metascraper = require('metascraper')([
  require('metascraper-author')(),
  require('metascraper-date')(),
  require('metascraper-description')(),
  require('metascraper-image')(),
  require('metascraper-logo')(),
  require('metascraper-publisher')(),
  require('metascraper-title')(),
  require('metascraper-url')()
])

Again, the order of rules are loaded are important: Just the first rule that resolve the value will be applied.

Use the first parameter to pass custom options specific per each rules bundle:

const metascraper = require('metascraper')([
  require('metascraper-logo')({
    filter: url => url.endsWith('.png')
  })
])

Rules bundles

?> Can't find the rules bundle that you want? Let's open an issue to create it.

Official

Rules bundles maintained by metascraper maintainers.

Core essential

• metascraper-audio – Get audio property from HTML markup.
• metascraper-author – Get author property from HTML markup.
• metascraper-citation – Get title, author, date, and publisher from Highwire Press citation_* and Dublin Core dc.* / dcterms.* tags.
• metascraper-date – Get date property from HTML markup.
• metascraper-defuddle – A Defuddle connector for metascraper.
• metascraper-description – Get description property from HTML markup.
• metascraper-feed – Get RSS/Atom feed URL from HTML markup.
• metascraper-feeds – Extract feed links (RSS/Atom/JSON) from HTML markup.
• metascraper-helpers – Common utility helpers for rule creation.
• metascraper-iframe – Get iframe for embedding content for the supported providers.
• metascraper-image – Get image property from HTML markup.
• metascraper-lang – Get lang property from HTML markup.
• metascraper-logo-bimi – Get logo property from the BIMI DNS record.
• metascraper-logo-favicon – Metascraper logo favicon fallback.
• metascraper-logo – Get logo property from HTML markup.
• metascraper-manifest – Metascraper integration for detecting PWA Web app manifests.
• metascraper-media-provider – Get specific video provider url (Facebook/Twitter/Vimeo/etc).
• metascraper-publisher – Get publisher property from HTML markup.
• metascraper-readability – A Mozilla readability connector for metascraper.
• metascraper-title – Get title property from HTML markup.
• metascraper-url – Get url property from HTML markup.
• metascraper-video – Get video property from HTML markup.

Vendor specific

• metascraper-amazon – Metascraper integration for Amazon.
• metascraper-bluesky – Metascraper integration for Bluesky.
• metascraper-dribbble – Metascraper integration for Dribble.
• metascraper-imdb – Metascraper integration for IMDb.
• metascraper-instagram – Metascraper integration for Instagram.
• metascraper-pinterest – Metascraper integration for Pinterest.
• metascraper-reddit – Metascraper integration for Reddit.
• metascraper-soundcloud – Metascraper integration for SoundCloud.
• metascraper-spotify – Metascraper integration for Spotify.
• metascraper-telegram – Metascraper integration for Telegram.
• metascraper-tiktok – Metascraper integration for TikTok.
• metascraper-uol – Metascraper integration for UOL URLs.
• metascraper-x – Metascraper integration for X (formerly Twitter)
• metascraper-youtube – Metascraper integration for YouTube.

Community

Rules bundles maintained by individual users.

• metascraper-address – Get schema.org formatted address.
• metascraper-shopping – Get product information from HTML markup on merchant websites.

See CONTRIBUTING to add your own module!

API

constructor(rules)

Create a new metascraper instance declaring the rules bundle to be used explicitly.

rules

Type: Array

The collection of rules bundle to be loaded.

metascraper(options)

Call the instance for extracting content based on rules bundle provided at the constructor.

options

html

Type: String

The HTML markup for extracting the content.

htmlDom

Type: object

The DOM representation of the HTML markup. When it's not provided, it's get from the html parameter.

omitPropNames

Type: Set Default: []

A set of property names that should be omitted. When specified, these properties will be missing in the returned metadata objects, and rules related to that will not be computed.

pickPropNames

Type: Set Default: undefined

A set of property names to pick for the metadata extraction process. When specified, only rules for these properties will be executed, and all other properties will be omitted. Takes precedence over omitPropNames when both are specified.

rules

Type: Array

You can pass additional rules to add on execution time.

These rules will be merged with your loaded rules at the beginning.

url

Required Type: String

The URL associated with the HTML markup.

It is used for resolve relative links that can be present in the HTML markup.

it can be used as fallback field for different rules as well.

validateUrl

Type: boolean Default: true

Ensure the URL provided is validated as a WHATWG URL API compliant.

const metascraper = require('metascraper')([
  require('metascraper-title')(),
  require('metascraper-image')(),
  require('metascraper-description')()
])

const html = '<title>Example</title><meta property="og:image" content="image.jpg">'
const url = 'https://example.com'

// Omit the image property
const omitPropNames = new Set(['image'])
const metadata = await metascraper({ url, html, omitPropNames })

console.log(metadata)
// Output: { title: 'Example', image: null, description: null }

Environment Variables

METASCRAPER_RE2

Type: boolean Default: true

It attemptt to load re2 to use instead of RegExp.

Benchmark

To demonstrate metascraper's exceptional accuracy, here is how it outperforms similar libraries:

Library	metascraper	html-metadata	node-metainspector	open-graph-scraper	unfluff
Correct	95.54%	74.56%	61.16%	66.52%	70.90%
Incorrect	1.79%	1.79%	0.89%	6.70%	10.27%
Missed	2.68%	23.67%	37.95%	26.34%	8.95%

A big part of the reason for metascraper's higher accuracy is that it relies on a series of fallbacks for each piece of metadata, instead of just looking for the most commonly-used, spec-compliant pieces of metadata, like Open Graph.

metascraper's default settings are targetted specifically at parsing online articles, which is why it's able to be more highly-tuned than the other libraries for that purpose.

If you're interested in the breakdown by individual pieces of metadata, check out the full comparison summary, or dive into the raw result data for each library.

License

metascraper © Microlink, released under the MIT License. Authored and maintained by Microlink with help from contributors.

microlink.io · GitHub microlinkhq · X @microlinkhq