PBS 75 of X — AJAX Intro
Matching Podcast Episode 589
Listen along to this instalment on episode 589 of the Chit Chat Across the Pond Podcast
You can also Download the MP3
PBS 74 Challenge Extension & Hint
I’ve decided to give everyone an extension on the challenge set at the end of the previous instalment.
Based on some listener feedback, I’d also like to give a little hint you might find helpful.
The JSON object is not intended to be used as the actual view object when rendering templates. It’s designed to be used as the data source from which you build your view objects.
Rather than look at all of HTTP in great detail, I’m going to focus this overview on the parts of the HTTP spec that are relevant to AJAX. If you’d like to know more, I’d suggest checking out instalments 34 and 35 of the Taming the Terminal series.
Starting with the very basics, HTTP is the Hyper-Text Transfer Protocol. Web browsers use this protocol to fetch data for URLs starting with
https://. HTTP is a client-server request-response protocol — clients (web browsers in our case) send HTTP requests to web servers, and web servers respond with HTTP responses. The protocol is stateless, so each request+response is seen as unrelated to every other request+response. In the modern world we’re accustomed to having web servers remember who we are from click to click, so how is that possible? Cookies were literally invented to work around the fact that HTTP is a stateless protocol. A cookie is simply a token a web server can include in an HTTP response that the client then inserts into all future requests so the web server can recognise you.
https:// URLs are equivalent.
At the very highest level, HTTP requests contain a URL, and HTTP responses contain the data at the requested URL, or some information about the data at the requested URL.
HTTP requests consist of:
- A URL
- A HTTP method
- Arbitrarily many HTTP Request Headers
- Optional form data
The HTTP method is used to specify the type of request the browser is making. For now, we’ll only be using HTTP to fetch information from web servers, but the protocol is much broader than that, and provides mechanisms for sending data insertion, data update, and even data deletion requests to servers.
URLs are surprisingly complex things, and they can contain a lot of information. For our purposes, URLs have the following components:
- A URL Scheme, or Protocol followed by the symbols
://. For our purposes, the scheme will always be one of
- A host, or server (if relevant), e.g.
podfeet.com. The host is appended to the scheme, but separated from it by the
filescheme omits the host part of the URL completely.
- A port number, implied or explicit (if appropriate). Explicit port numbers are appended to the host, separated from it with the
:symbol. URL schemes imply port numbers where appropriate, so you can usually omit them.
filescheme omits port numbers completely.
- A path or file path which specifies a location on a host. The file path always starts with a
- An optional page fragment which specifies a named location within a page. If present, the fragment is appended after the file path, separated from it with a
- An optional query string which specifies an arbitrary number of name-value pairs known as query parameters. If present, the query string is appended after the page fragment (or file path if there is no page fragment), and separate from it with the
?symbol. Each name-value pair is separated from the others with an
&symbol, and the names are separated from the values with an
Note that special characters have to be escaped in URLs. They’re escaped using the
Let’s tie all that together with an example URL:
We can then say the following:
- The scheme is
- The host is
- The port is
- The file path is
- The page fragment is
- The query string is
- There are. two query parameters;
- The value of the query parameter
- The value of the query parameter
%20is the URL escape code for the space symbol).
For now, there are just two HTTP methods relevant to us —
GET requests should never alter the internal state of the server, and it should be safe to cache the responses.
POST requests should be used when the intent of the request is to change the server’s state in some way (e.g. submit a comment on a blog post), and the responses to
POST requests should not be cached.
From a practical point of view, the biggest difference between
POST is how form data gets encoded into the request.
POST requests can include relatively small amounts of data in the form of query parameters within the URL. In addition to that,
POST requests can contain as much data as is needed in the so-called request body, but
GET requests can’t.
It’s very important to note that query parameters should never be used to send sensitive data because they are a part of the URL, and URLs get logged. For this reason, always send sensitive data to web servers with
GET is the default method, and that’s what we’ll be using for now.
HTTP Request Headers and Cookies
HTTP request headers allow the client to share certain defined pieces of information with the server. The most widely used request header is the
User-Agent header which the browser uses to identify itself to the server.
Cookies are actually passed to the server using the
Cookie HTTP header.
Seeing an HTTP Request
To help you see the innards of an HTTP request, I’ve created a little server-side script that echoes back details of the HTTP request it received in an HTTP response: https://bartb.ie/utils/httpEcho/.
As well as seeing the request in HTML, you can also see it in pure JSON format at https://bartb.ie/utils/httpEcho/json, in pretty-printed JSON format at https://bartb.ie/utils/httpEcho/jsonText, and in plain text (Markdown really) at https://bartb.ie/utils/httpEcho/text.
Note that you can pass query parameters to the script by adding them to the end of the URL, e.g. https://bartb.ie/utils/httpEcho/?param1=boogers¶m2=big%20snot
Note that you can find the full source code for these PHP scripts on GitHub. Remember that one of the reasons I gave for loving Mustache is that it’s available in so many languages. Well, to underline that point, note that the HTML and plain-text views are both generated using Mustache templates, and that the view object used is the JSON object presented by the JSON and pretty-printed JSON scripts.
A HTTP response consists of the following:
- An HTTP status code
- Arbitrarily many HTTP response headers
- If appropriate, a response body containing the requested data
Servers use HTTP status codes to indicate the nature of their response. These are always 3-digit numbers, and can be accompanied by a short human-readable title/description. The one you want is
Under the hood HTTP status codes are grouped into categories by leading digit:
- The browser deals with informational responses for us, so we don’t have to worry about them. Examples include
100 Continue, and
101 Switching Protocols.
POSTrequests, the response code you always hope to get is
200 OK. That means the request was successfully processed, and that the response contains the requested data. There are other successful status codes for other HTTP methods, e.g.
301 Moved Permanentlyfor permanent redirects, and
302 Foundfor temporary redirects. The main difference is that browsers are permitted to cache permanent redirects, but temporary redirects should be checked every time in case they’ve changed.
4**— Client Errors
400 Bad Request. However, there are two client errors we are likely to encounter from time to time — the infamous
404 Not Found, and the frustrating
5**— Server Errors
- Later in the series we’ll be moving into server-side coding, at which time we’ll almost certainly get to see lots of
500 Internal Server Errorresponses. For now, any server error status codes indicate problems beyond our control.
404 Not Found, and
500 Internal Server Error.
When it comes to processing HTTP responses, we generally just need to deal with two cases — we got a
200 OK, so our request was successful or we got any other status code, our request was not successful, and we need to handle that fact in some way.
Basically, anything other than
200 OK means there was some kind of error, and our code needs to deal with that in some way.
The HTTP response headers are used by the server to send information about the response back to the client. The most important HTTP response header is
Content-Type which tells the browser the MIME Type of the data included in the response. The server can also use the headers to give cookies to the browser, and to tell the browser how long it should cache the response for.
A Final Illustration
We can use my
httpEcho script in conjunction with the
curl terminal command (Mac & Linux only, I’m afraid) to peer inside a full HTTP transaction.
The command we’ll be using is:
curl -v --data-urlencode d1=val1 --data-urlencode d2=val2 --cookie 'c1=cookie; c2=monster' https://www.bartbusschots.ie/utils/httpEcho/text?p1=param1Val\&p2=param2Val
There’s a lot there, so let’s break it down piece-by-piece.
-v flag puts
curl into verbose mode. That means it will show us the raw HTTP request it is generating, as well as all the details of the HTTP response it receives.
--data-urlencode flags put
POST mode and add two pieces of form data named
d2 with the values
--cookie flag sets two cookies named
c2 with the values
Finally, we are calling the
httpEcho URL for the plain-text version of the output with a query string that specifies two query string parameters named
p2 with the values
When we run the command we get a lot of output!
bart-imac2018:~ bart$ curl -v --data-urlencode d1=val1 --data-urlencode d2=val2 --cookie 'c1=cookie; c2=monster' https://www.bartbusschots.ie/utils/httpEcho/text?p1=param1Val\&p2=param2Val * Trying 18.104.22.168... * TCP_NODELAY set * Connected to www.bartbusschots.ie (22.214.171.124) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: CN=bartbusschots.ie * start date: Mar 5 08:15:48 2019 GMT * expire date: Jun 3 08:15:48 2019 GMT * subjectAltName: host "www.bartbusschots.ie" matched cert's "www.bartbusschots.ie" * issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 * SSL certificate verify ok. > POST /utils/httpEcho/text?p1=param1Val&p2=param2Val HTTP/1.1 > Host: www.bartbusschots.ie > User-Agent: curl/7.54.0 > Accept: */* > Cookie: c1=cookie; c2=monster > Content-Length: 15 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 15 out of 15 bytes < HTTP/1.1 200 OK < Server: nginx/1.12.2 < Date: Sat, 06 Apr 2019 15:19:04 GMT < Content-Type: text/plain;charset=UTF-8 < Transfer-Encoding: chunked < Connection: keep-alive < X-Powered-By: PHP/5.6.39 < # Client - IP: 126.96.36.199 - Browser (User Agent String): curl/7.54.0 # HTTP Request - URL: https://www.bartbusschots.ie/utils/httpEcho/text?p1=param1Val&p2=param2Val - Protocol Version: HTTP/1.1 - Method: POST - Headers (6): * Host: www.bartbusschots.ie * User-Agent: curl/7.54.0 * Accept: */* * Cookie: c1=cookie; c2=monster * Content-Length: 15 * Content-Type: application/x-www-form-urlencoded - Query String: p1=param1Val&p2=param2Val - Query Parameters (2): * p1: param1Val * p2: param2Val - Form Data (2): * d1: val1 * d2: val2 - Cookies (2): * c1: cookie * c2: monster # Server - IP:Port: 188.8.131.52:443 - Name: bartbusschots.ie - Software: nginx/1.12.2 - CGI Revision: CGI/1.1 * Connection #0 to host www.bartbusschots.ie left intact
Again, let’s break it down.
The first thing we see is some information about what
curl is doing. You see it trying to connect to the IP address of my server, you see it succeeding on port 443, and then you see it successfully negotiate a secure connection using TLS.
At this stage
curl is finally ready to send the HTTP request, so the next thing we see is that request:
> POST /utils/httpEcho/text?p1=param1Val&p2=param2Val HTTP/1.1 > Host: www.bartbusschots.ie > User-Agent: curl/7.54.0 > Accept: */* > Cookie: c1=cookie; c2=monster > Content-Length: 15 > Content-Type: application/x-www-form-urlencoded
curl uses the
User-Agent header to identify itself, and the
Cookie header to pass our two cookies to the server.
The next thing we see is the HTTP response from the servers, starting with the status code and the response headers:
< HTTP/1.1 200 OK < Server: nginx/1.12.2 < Date: Sat, 06 Apr 2019 15:19:04 GMT < Content-Type: text/plain;charset=UTF-8 < Transfer-Encoding: chunked < Connection: keep-alive < X-Powered-By: PHP/5.6.39
Notice we got a
200 OK status, and the use of the
Content-Type HTTP response header to specify both the MIME Type and the text encoding of the returned data.
Finally, we see the actual data returned by the server, in this case, the plain-text echoing of the data the server received just as it would appear in a browser window:
# Client - IP: 184.108.40.206 - Browser (User Agent String): curl/7.54.0 # HTTP Request - URL: https://www.bartbusschots.ie/utils/httpEcho/text?p1=param1Val&p2=param2Val - Protocol Version: HTTP/1.1 - Method: POST - Headers (6): * Host: www.bartbusschots.ie * User-Agent: curl/7.54.0 * Accept: */* * Cookie: c1=cookie; c2=monster * Content-Length: 15 * Content-Type: application/x-www-form-urlencoded - Query String: p1=param1Val&p2=param2Val - Query Parameters (2): * p1: param1Val * p2: param2Val - Form Data (2): * d1: val1 * d2: val2 - Cookies (2): * c1: cookie * c2: monster # Server - IP:Port: 220.127.116.11:443 - Name: bartbusschots.ie - Software: nginx/1.12.2 - CGI Revision: CGI/1.1
$.ajax() function to make HTTP requests. That’s where we’ll start the next instalment.
Join the Community
Find us in the PBS channel on the Podfeet Slack.