For website owners
SpaceJobsNowBot: the crawler behind SpaceJobs.Now
SpaceJobs.Now gathers open roles across the space industry, from launch providers to satellite makers, into one free board. Once a night its crawler visits the openings companies publish, and every listing names the company and the posting it came from.
Rules
What the crawler does and the lines it never crosses
What it collects
Openings that space and aerospace companies post in public: their careers pages, the applicant tracking systems behind them, and job boards that syndicate them. SpaceJobs.Now lists each opening under the company that posted it, with the posting's own words. Pages behind a login, a paywall or a human check are left alone, and the crawler never fills in, submits or applies to anything.
How you can recognise it
It sends the user agent "SpaceJobsNowBot/1.0 (+https://spacejobs.now; space industry job aggregator)" with every request, and never borrows the name of a browser or of another company's crawler.
How it behaves
Before anything else it fetches robots.txt and skips every path closed to SpaceJobsNowBot or to crawlers in general. Where a site sets a Crawl-delay it waits that long between requests, it keeps only a handful of requests open to one host, slows down on a "too many requests" answer and gives up on a host that goes on refusing. Every company's openings are collected once a night, between 05:00 and 07:00 UTC; the page of a single job is fetched only when the job first appears or drops off the company's list.
Contact
Questions about the crawler
If you want your openings collected less often, differently or not at all, put it in your robots.txt (we fetch it again each night) or email us; every message gets an answer. The About page says more about SpaceJobs.Now.