metaltoad.com - Exclude crawlers server-wide with X-Robots-Tag
Mi. 01.06.2011 - 18:26
For a staging site, it's important to exclude crawlers. You wouldn't want your content to get indexed at the wrong URL! The conventional wisdom is to use HTTP Basic authentication. There are some disadvantages to this approach however, and I've found I prefer using a new HTTP header called X-Robots-Tag. Note that this assumes your only objective is to prevent indexing by benevolent crawlers…
More news from metaltoad.com
06.05.2020
Metal Toad’s AWS Media Services Hackathon
Corinna Gelster-Borgardt
Wed, 05/06/2020 - 19:07
This is an…
07.04.2020
The Value of Values
Jonathan Mills
Tue, 04/07/2020 - 20:26
Given the times it seems appropriate to lean back into some of the…
02.04.2020
Cloud Means Business: Executive Interview With Carmen V. Lidz
Joaquin Lippincott
Thu, 04/02/2020 - 00:52…
31.03.2020
Cloud Means Business: Executive Interview With Bill Martin
Joaquin Lippincott
Mon, 03/30/2020 - 22:55
As…
25.03.2020
Cloud Means Business: Executive Interview With Bridget Barnes
Joaquin Lippincott
Wed, 03/25/2020 - 22:40…
23.03.2020
Cloud Means Business: Executive Interview with Charles Eckstrom
Joaquin Lippincott
Mon, 03/23/2020 - 21:22…
20.03.2020
Cloud May be ITs Antidote for COVID-19
Joaquin Lippincott
Fri, 03/20/2020 - 18:09
These are…
18.03.2020
Cloud Means Business: Executive Interview with Jim Williamson
Joaquin Lippincott
Wed, 03/18/2020 - 18:20…
12.03.2020
The Impact of COVID-19 on the entertainment…
10.03.2020
Creating a Scavenger Hunt on Android With TensorFlow
James Simshaw
Tue, 03/10/2020 - 21:53
While going through my files sorting…
- 1 von 67
- ››