3

Uma dúvida conceitual (ou não):

Dos métodos HTTP, quais deles não podem ser "rastreados" - ou interpretados - por um crawler?

  • POST
  • GET
  • PUT
  • PATCH
  • DELETE

Alguém com conhecimento no assunto pode nos responder?

2 Respostas 2

0

Pela teoria, crawlers costumam executar métodos idempotentes e seguros - OPTIONS, GET, HEAD.

Do livro "Cloud Standards: Agreements That Hold Together Clouds": "Web crawlers, for example, use only safe methods to avoid disturbing data on the sites they crawl"

Ou ainda: "Rastreadores da Web, por exemplo, utilizar apenas métodos seguros para evitar dados inquietantes sobre os sites que rastejam"

O que faz todo o sentido para o propósito de um crawler, se pensarmos logicamente.

Uma ótima referência sobre o assunto é https://www.whitehatsec.com/blog/http-methods/

Idempotency and safety are important attributes of HTTP methods. An idempotent request can be called repeatedly with the same results as if it only had been executed once. If a user clicks a thumbnail of a cat picture and every click of the picture returns the same big cat picture, that HTTP request is idempotent. Non-idempotent requests can change each time they are called. So if a user clicks to post a comment, and each click produces a new comment, that is a non-idempotent request.

Safe requests are requests that don’t alter a resource; non-safe requests have the ability to change a resource. For example, a user posting a comment is using a non-safe request, because the user is changing some resource on the web page; however, the user clicking the cat thumbnail is a safe request, because clicking the cat picture does not change the resource on the server.

Production safe crawlers consider certain methods as always safe and idempotent, e.g. GET requests. Consequently, crawlers will send GET requests arbitrarily without worrying about the effect of repeated requests or that the request might change the resource. However, safe crawlers will recognize other methods, e.g. POST requests, as non-idempotent and unsafe. So, good web crawlers won’t send POST requests.

RFC sobre Safe e Idempotent Methods: http://w3.org/Protocols/rfc2616/rfc2616-sec9.html

9.1.1 Safe Methods

Implementors should be aware that the software represents the user in their interactions over the Internet, and should be careful to allow the user to be aware of any actions they might take which may have an unexpected significance to themselves or others.

In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.

Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them.

9.1.2 Idempotent Methods

Methods can also have the property of "idempotence" in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request. The methods GET, HEAD, PUT and DELETE share this property. Also, the methods OPTIONS and TRACE SHOULD NOT have side effects, and so are inherently idempotent.

However, it is possible that a sequence of several requests is non- idempotent, even if all of the methods executed in that sequence are idempotent. (A sequence is idempotent if a single execution of the entire sequence always yields a result that is not changed by a reexecution of all, or part, of that sequence.) For example, a sequence is non-idempotent if its result depends on a value that is later modified in the same sequence.

A sequence that never has side effects is idempotent, by definition (provided that no concurrent operations are being executed on the same set of resources).

17
  • 1
    Eu acho que se colocar mais um bloco embaixo traduzindo o termo em amarelo ajuda. Do jeito que está, não faz muito sentido no SOpt.
    – Largato
    23/03/2016 às 20:53
  • 1
    Infelizmente, apesar do stack ser em português, o mundo em que vivemos está repleto de conteúdo em inglês. Isso implica em, mesmo que não gostemos, aprender a conviver com o idioma. 23/03/2016 às 20:54
  • 2
    O site é em português, logo, é natural que esperemos reposta em português. Você nao pode supor que TODO usuário que participe aqui, é obrigado a entender inglês. Eu mesmo não sei inglês e a resposta não me ajudaria se eu tivesse com um problema semelhante.
    – user28595
    23/03/2016 às 21:00
  • 1
    Sintam-se livres para traduzir o conteúdo, tal como expressões técnicas, crawler, put, get, options, head e também o RFC, se for o caso. 23/03/2016 às 21:04
  • 1
    @IsvaldoFernandes Não precisa traduzir "na integra" a citação inteira, mas fazer uma breve explicação(não precisa ser uma redação do ENEM também) sobre o que a citação diz, já torna a resposta muito mais completa. Entendo as dificuldades em se traduzir coisas técnicas, mas um resumo já ajuda quem lê a ter uma ideia.
    – user28595
    23/03/2016 às 21:11
0

Isso independe do Crawler, você pode simular qualquer request.

Curl

 curl --request POST 'http://www.somedomain.com/'
 curl --request DELETE 'http://www.somedomain.com/'
 curl --request PUT 'http://www.somedomain.com/'

fonte: Link

Python

r = requests.put("http://httpbin.org/put")
>>> r = requests.delete("http://httpbin.org/delete")
>>> r = requests.head("http://httpbin.org/get")
>>> r = requests.options("http://httpbin.org/get")

fonte: Link

Java

GetRequest request = Unirest.get(String url);
GetRequest request = Unirest.head(String url);
HttpRequestWithBody request = Unirest.post(String url);
HttpRequestWithBody request = Unirest.put(String url);
HttpRequestWithBody request = Unirest.patch(String url);
HttpRequestWithBody request = Unirest.options(String url);
HttpRequestWithBody request = Unirest.delete(String url);

Lib:Link

2
  • Amigo, a questão é saber qual método http (se existe) um crawler não pode rastrear... 23/03/2016 às 20:56
  • @MarllonNasser, a resposta foi clara, não existe. 24/03/2016 às 0:04

Você deve fazer log-in para responder a esta pergunta.

Esta não é a resposta que você está procurando? Pesquise outras perguntas com a tag .