Silicon Soul

The first step in data communication is to transform digital information inside of a computer into something that is transmittable to another computer, and to transform that information back to its original form at the other end of the communication channel.

Note: If you are not familiar with the GNU/Linux command line interface, or you intend to use a script obtained from this site, review the Conventions page before proceeding.

Networks

Data networks can be roughly divided into two groups:

1. Local area networks (LANs) connect a small number of nodes in a geographically limited area.
2. Wide area networks (WANs) connect a potentially large number of nodes in a geographically large area.

There are many different networking technologies for diverse requirements.

A protocol is an agreed upon scheme governing how two (or more) nodes on a network talk to one another. It is a common networking language that must be configured for network hosts to communicate. The spectrum of possible protocols ranges from rules for electrical signals on an Ethernet cable or radio signals in a Wireless LAN (WLAN), up to protocols governing access to an SQL database server.

In a protocol stack, we try to separate the individual components of the stack, i.e., the electrical part, basic communication between computers on the same network, basic communication between computers on different networks, and services like electronic mail (email) or the World Wide Web (WWW).

Protocols

Protocols can be roughly divided into three classes:

1. Transmission Protocols
2. Communication Protocols
3. Application Protocols

Transmission Protocols

Transmission protocols (also referred to as access methods) govern data transmission at the level of network cards and physical connections. Their make-up depends on the physical properties and restrictions arising from their implementation in hardware.

The most common transmission protocol in LANs is Ethernet. There are other standards for special applications, e.g., WLAN access methods, like IEEE 802.11ac.

Communication Protocols

Communication protocols serve to organize the communication between computers in different networks without presupposing detailed knowledge of the medium access methods (i.e., the physical infrastructure) used. Communication protocols are supposed to prevent you from having to deal with transmission protocols, but they cannot exist without them. Essentially, communication protocols abstract away transmission protocol details.

Application Protocols

Application protocols implement actual services like email, file transfer, or Internet telephony based on communication protocols. Application protocols let you make sense of the data that communication protocols send and receive.

Protocol Data Units

The data exchanged via a protocol are called protocol data units. Depending on the protocol used, they may have more specific names, e.g., frames (IEEE 802.3/Ethernet), packets (ICMP), datagrams (IP), or segments (TCP).

The Open Systems Interconnection (OSI) Model

The fact that communication protocols are meant to obscure the details of transmission protocols, and that application protocols are meant to obscure the details of communication protocols lets us construct a layer model. The most well-known layer model is the Open Systems Interconnection (OSI) model.

Layers 1 and 2 (Physical and Data Link layers) describe how data is sent on the physical infrastructure. This includes the medium access method (Layer 1), as well as the encoding of the data (Layer 2).

Layer 3 (the Network layer) defines the functions required for routing, including requisite addressing. Layer 4 (the Transport layer) describes the transport of application data and distinguishes between connection-oriented (e.g., TCP) and connection-less (e.g., UDP) services.

Layers 5, 6, and 7 (Session, Presentation, and Application layers, respectively) describe the system-independent representation of data within the network and the interfaces to application protocols.

The transmission protocols take up the lowest layers of the model and the application protocols take up the highest layers. Every layer on the sender's side receives data from a layer above and passes it to a layer below. On the receiver's side, data is received from a layer below and passed on to a layer above.

Information exchange across networks requires packets. Packet header information describes where the packet came from, where it is going, and where it fits in the sequence of packets that constitute the network stream.

Within each layer on the sender side, the corresponding protocol receives a protocol data unit at its service interface from the layer above and adds a header containing all of the information important for its operation before passing it on across the service interface of the layer below. The layer below considers everything it receives on the service interface as data. The previous protocol's header is of no concern to the lower layer.

On the receiving side, data units pass through the same layers in reverse order, and every layer removes its header before passing the payload upwards.

Many protocol stacks cannot be directly mapped to the OSI model. Also, the OSI model is not a binding standard. It is a reference model that clarifies the concepts involved and makes them easier to discuss.

Transmission Control Protocol/Internet Protocol (TCP/IP)

Transmission Control Protocol/Internet Protocol (TCP/IP) is the most wide-spread method of transferring data in computer networks. TCP/IP is not just a single protocol, but a plethora of different protocols built upon one another with different applications. This is called a protocol family (a protocol stack is an implementation of a protocol family).

The Internet Protocol (IP) works in conjunction with TCP/UDP to fragment, transmit, defragment, and resequence network data, while Internet Control Message Protocol (ICMP) is used to test and verify network communications between hosts.

These are some of the most important protocols from the TCP/IP protocol family:

Transmission Protocols

Layer 1 (Physical) - Ethernet, WLAN, DSL, Bluetooth
Layer 2 (Data Link) - PPP, ARP, L2TP, MAC

Communication Protocols

Layer 3 (Network) - IP, ICMP, IPsec
Layer 4 (Transport) - TCP, UDP

Application Protocols

Layer 5 (Session) - Named Pipe, SOCKS, PPTP, RTP, NetBIOS
Layer 6 (Presentation) - MIME, XDR
Layer 7 (Application) - HTTP, DHCP, DNS, FTP, SSH, SIP

For a more complete view of the protocols that the TCP/IP protocol family is composed of, view the contents of the /etc/services and /etc/protocols files on your GNU/Linux system. Keep in mind, few services use the /etc/services file. Most services have settings in their configuration files that specify which port to listen to for their respective service.

Internet Protocol (IP)

IP connects two nodes. As an OSI layer 3 protocol, it is responsible for ensuring that data finds its way across the Internet from the sender to the receiver. This path can involve long distances consisting of diverse, independent sections using distinct networking technologies and exhibiting varying communications parameters.

Different parts of the route use not only distinct networking technologies, but also different local addresses (or no required addressing at all when using protocols like the Point-to-Point Protocol, PPP, i.e., there are only two communication stations). IP makes available a global address space that assigns a unique address to every node connected to the Internet, by which that node can be identified.

IP is a connection-less protocol. Unlike the traditional telephony network, IP provides no fixed connection (i.e., a wire) for two systems to communicate. The data to be transmitted is divided up into small pieces (i.e., the IP datagrams), which can then be addressed and delivered independently from each other.

In principle, every datagram can take a different path to the receiver than the previous one. This makes IP resilient to failure of connections or routers, as long as one route can be found from the source to the target node.

IP does not guarantee that all transmitted data will reach the receiving system, nor does it guarantee that the data that does arrive will do so in the order in which it was sent. It is up to higher-level protocols to sort this out if the receiving application requires it.

IP takes care of fragmentation and ensures that only datagrams with no missing fragments are officially considered received. Fundamentally, IP allows datagrams of up to 65,535 bytes, but not every medium access method can actually transmit these datagrams in one piece.

For example, Ethernet allows frames of at most 1,518 bytes, including 14 bytes for the frame header and 4 bytes for a checksum at the end of the frame, that is, an Ethernet frame can carry at most 1,500 bytes of payload, and if the IP layer above it wants to transmit a larger datagram, that datagram must be fragmented (i.e., split across several frames). We say that the maximum transmission unit (MTU) for Ethernet is 1500 bytes.

The IP implementation of the sending node cannot foresee which medium access method will be used on the way to the destination node and whether fragmentation will be required (and if so, how large the fragments may be). This only becomes known when data is actually transmitted.

Instead, datagrams are typically sent with the don't fragment (DF) bit in the header switched on, which forbids other routers from breaking them up further. If such a datagram arrives at a point where it is too big for the next hop, the router in question uses ICMP to send a destination unreachable - fragmentation needed message. In this case, the sending node can try again using smaller fragments. This method is called path MTU discovery (PMTUD).

This process can go wrong, i.e., if an over aggressive firewall blocks ICMP traffic. In this case, the error messages concerning the required MTU never reach the sender of the datagrams, who consequently has no idea what is happening.

In practice, this leads to web pages not being correctly displayed, and/or connections that simply hang. Normally, the problems disappear if you manually set the MTU for the interface in question to 1492. Then, the remote node adheres to that value.

The time to live (TTL) states the maximum life span of the datagram. It is set by the sender and decremented (reduced by 1) by each node the datagram passes through on its way to the recipient. If the TTL reaches zero, the datagram is dropped and the sender is notified. This serves to prevent meandering datagrams that, due to routing errors, run in circles on the Internet without ever reaching their destination. A common default TTL value is 64.

The type of service (TOS) specifies the quality of service desired for the datagram. Theoretically, you can choose (in addition to one of seven precedence levels, which will be ignored) any of the attributes low latency, high throughput, high reliability, or low cost. These options are only advisory and routers usually ignore them.

If a problem or interruption develops during connection-less transmission, the network can look for alternative routes for future datagrams that detour around the damaged part. Methods like TCP make it possible to detect which data was lost due to the problem (i.e., error-checking) and arrange for it to be re-transmitted.

Internet Protocol (IP) and Transmission Protocols

Every Ethernet interface has a unique address, the 48-bit Media Access Control (MAC) address, and all nodes are connected (if only conceptually) to a shared medium (e.g., a segment). Everything a node sends is received by all other nodes, but these nodes usually only accept those protocol data units that are addressed to them. If two nodes simultaneously transmit, a collision occurs, which is handled by both nodes stopping transmission, waiting for a random period of time, and trying again.

Ethernet protocol data units, the frames, can be sent either to particular other nodes within the segment by specifying their MAC address as the recipient (the frame will be seen by all nodes, but ignored by all except for the addressed node), or broadcast to all nodes on the segment.

Ethernet adapters also support a promiscuous mode, in which all frames, even ones that would otherwise be ignored, are passed to the operating system. This is used for things like network analysis programs and packet analyzers (e.g., Wireshark).

If a node (i.e., node A) wants to communicate with another node (i.e., node B), whose IP address it knows, but whose MAC address it does not know, it asks all connected nodes by Ethernet broadcast who is assigned the IP address in question. When a positive response is achieved, it receives the MAC address of the responder.

This procedure follows the Address Resolution Protocol (ARP). Once node A has received node B's MAC address, it stores it for a certain time in its ARP cache, in order to not have to repeat the query for every frame.

IP datagrams to nodes whose IP and MAC addresses are part of the ARP cache can be directly addressed at the Ethernet level by embedding them as payload data into Ethernet frames. IP datagrams addressed to IP addresses that do not belong to nodes on the same Ethernet segment must be routed.

You can access the ARP cache using the arp command, not just to read, but also to write new entries.

Internet Protocol Version 4 (IPv4) Networking

Every network interface (device) in a system on a TCP/IP network has at least one IP address. Here, an interface is the part of a system that is able to send and receive IP datagrams. A single system can contain more than one such interface (and then generally uses more than one IP address). With the ip address show command, you can list the configured network devices.

IP Version 4 (IPv4) addresses are 32 bits long and are usually written as dotted quads. They are viewed as a sequence of four eight-bit numbers written in decimal notation as values between 0 and 255, e.g., 203.177.8.4.

Each IP address is assigned to be globally unique and denotes a node in a particular network and a host part. This split between the network part (i.e., the network prefix) and host part (i.e., the host identifier) of an IP address is variable and can be adapted to the number of node addresses required in a network.

If the host part takes n bits, 32-n bits remain for the network part. The split between network and host is documented by the netmask, which contains a binary 1 for each bit in the IP address belonging to the network prefix and a binary 0 for each bit of the host identifier. The netmask is notated either as a dotted quad or, frequently, as the number of binary ones (i.e., the length of the network prefix).

For example, 203.177.8.4/24 is an IP address in a network with a netmask of 255.255.255.0 in dotted quad decimal notation, or 11111111.11111111.11111111.00000000 (i.e., /24) in binary notation. For two hosts on the same network to communicate, they need to have the same network prefix (and the same netmask).

Traditionally, the set of IP addresses from 0.0.0.0 to 255.255.255.255 was divided into several network classes, which were called class A, class B, class C, class D (multicast addresses), and class E (experimental addresses). Now, IP address allocation and routing uses Classless Inter-Domain Routing (CIDR).

CIDR makes it possible to assign arbitrary netmasks for subnets (i.e., subnet masks) and makes it possible to fine-tune the sizes of address ranges made available to customers (usually Internet Service Providers, ISPs), as well as work against the increasing size of routing tables.

By convention, the first and last IP addresses in a network are reserved for special purposes. The first address (host part all binary zeros) is the network address. The last address (host part all binary ones) is the broadcast address.

For example, if you have a network with the network address of 203.177.8.0 and a netmask of 255.255.255.224 (i.e.,11111111.11111111.11111111.11100000 or /27), the broadcast address would be 203.177.8.31. IP addresses from 203.177.8.1 through 203.177.8.30 would be available for nodes (i.e., 30 nodes are supported).

You can determine how many nodes a network address can support by taking 2 to the number of binary zeroes in the network's netmask, minus 2 (for the network address and broadcast address):

Number of nodes = (2^Number of netmask binary zeroes) - 2

The address 255.255.255.255 is a broadcast address for the local network segment. This address is used if no more precise address is known, e.g., if a node wants to obtain an IP address and netmask via DHCP.

Routing is used to send IP datagrams that cannot be directly delivered within the local network on to the correct destination, i.e., routing applies where the recipient of an IP datagram cannot be found within the same network as the sender. The sender can determine this based on the desired recipient's IP address, by considering the part of the destination address that is covered by its own netmask and checking whether this matches its own network address. If this is the case, the recipient is local and can be directly reached.

If a recipient cannot be directly reached, the node consults a routing table that should contain at least a default gateway, i.e., a node that takes care of forwarding IP datagrams that cannot be delivered outright (usually, this node needs to be directly reachable). Such a node is called a router and is either a computer or a special appliance manufactured for this purpose.

The router contains various network interfaces, each of which is assigned an IP address and a netmask, and can deliver IP datagrams immediately to nodes that, according to the netmasks of its interfaces, can be identified as being part of one of its networks. If necessary, other directly reachable nodes acting as routers are called upon for more forwarding.

Routing tables can be complex, e.g., it is possible to forward IP datagrams directed to particular nodes or networks to other routers that are not the default gateway.

Usually, a node (computer or router) only determines the directly following routing step (i.e., a hop), instead of specifying the complete path from the original sender of the IP datagram to the final recipient. This means that it is up to each router between the sender and recipient to pick a hop that it considers optimal. Well-configured routers talk to their neighbors and can base their routing decisions on information about network load and potential blockages elsewhere in the network.

It is possible for an IP datagram to specify the complete path it wants to take to a destination. This is called source routing. It is universally discouraged and will be completely ignored by large parts of the network infrastructure. It is discouraged because it is at odds with the idea of dynamic load distribution and is a common vehicle for security issues.

IPv4 Private Networks and the Loopback Address

The following IPv4 address ranges are reserved for private networks:

It is common for these addresses to be used for LANs.

The entire 127.0.0.0/8 range is for loopback networking only. It has 16,777,216 addresses. The special hostname localhost is associated with the IPv4 address 127.0.0.1.

Subnets

A physical grouping of hosts attached to the same wire, hub, or switch is a segment (e.g., a shared Ethernet medium). A logical grouping of hosts on a network is a subnet.

Frequently, a large network is too imprecise. Therefore, operators often divide their networks into several smaller networks. This is referred to as subnetting and is accomplished by extending the network prefix portion of the netmask.

Above, an address space of 150.150.0.0/16 is provided with the intent of creating subnets from that space that support at least 50 host addresses each.

To find the number of bits that we have to work with to create our new subnet mask from the given IP address, we can subtract the sum of the number of reserved bits and network prefix length (i.e., N + NID) from the total length of the IP address (L).

L - (N + NID) = SHID
32 - (2 + 14) = 16 bits

To determine how many bits the host portion of our new subnet mask may contain (i.e., the HID_min), we can take the Log Base 2 of the number of host addresses each subnet should have (i.e., 50) plus2, and round the value up:

|log_2(Minimum number of hosts per subnet + 2)| = HID_min  
|log_2(50 + 2)| = |5.7| = 6 bits

Now that we know that the host portion of our new subnet mask should be at least 6 bits, we can determine the maximum number of bits that we should extend the network prefix of the netmask to create our subnets (i.e., SID_max).

SHID - HID_min = SID_max  
16 - 6 = 10 bits

For IPv4, 1 bit is used for SID_min, and 10 bits has been determined to be our SID_max. From these options, we choose a SID of 10 bits.

From our SID value, we can determine the number of subnets we will create:

2^SID
2^10 = 1024 subnets

Next, we can finalize the number of bits to use for the host portion of our new subnet mask (i.e., HID):

SHID - SID = HID  
16 - 10 = 6 bits

Then, we can determine the total length that the network prefix of our new subnet mask should be:

N + NID + SID = /mask  
2 + 14 + 10 = /26

Our new subnet mask is /26, or in dotted octet decimal form, 255.255.255.192. We can start our subnets at 150.150.0.0 and increment our subnets by the Least Significant Bit (LSB) of our new subnet mask in binary form.

For example, 255.255.255.192 in binary form is 11111111.11111111.11111111.11000000. The LSB is the 1 furthest to the right. Take the octet that contains the LSB and convert any 1s before the LSB to 0s. This leaves you with the binary value of 01000000. Convert this value to decimal form and you have the value to increment your subnets by (i.e., 64).

Finally, we can define our subnets:

1st Subnet
Network Address: 150.150.0.0
Lowest Usable Address: 150.150.0.1
Highest Usable Address: 150.150.0.62
Broadcast Address: 150.150.0.63

Until...

1024th Subnet
Network Address: 150.150.255.192
Lowest Usable Address: 150.150.255.193
Highest Usable Address: 150.150.255.254
Broadcast Address: 150.150.255.255

Ultimately, we end up with 1024 subnets that support 62 hosts each (for each 64 block subnet, the first address is used for the network address and the last address is used for the broadcast address).

Network Address Translation (NAT)

IPv4 addresses are a scarce resource. Therefore, it is highly probable that you will be assigned only one public IP address by your ISP to connect all of your network devices to the Internet. The solution to connecting all of your devices with only a single IP address consists of masquerading, as well as port forwarding.

Both approaches are based on the fact that only your router is connected to the Internet by means of a public IP address. All other nodes within your network use addresses according to IPv4 private networks.

Masquerading implies that your router rewrites IP datagrams that nodes within your network send outside of your network in order to replace those nodes' IP addresses with its own, and forwards the corresponding response IP datagrams to the proper senders. Both the nodes inside of your network and the Internet are unaware of this fact, i.e., the former assume that they are directly talking to the Internet, while the latter only gets to see the official IP address of your router.

Conversely, port forwarding enables nodes on the Internet to connect to services like BitTorrent or HTTP through their respective ports on your router, while the router forwards the IP datagrams associated with those services to a node on the inside of your network that performs the actual service.

Masquerading and port forwarding are two examples of a concept called Network Address Translation (NAT). Specifically, masquerading can be thought of as source NAT, since the sender address of outgoing IP datagrams is modified, while port forwarding can be thought of as destination NAT, since the destination address of IP datagrams addressed to the receiver is changed.

The ip Command

The ip command can be used to set up both network interfaces and routes. It is the successor to both the ifconfig and route commands. Changes made with this command are not persistent and will be lost after a system reboot.

ip's syntax is roughly:

ip ex_options ex_object ex_command ex_parameters

ex_object can include:

• link Parameters of a network interface.
• address Protocol address (IPv4 or IPv6) of a network interface.
• route Routing table entry. Used for querying, adding, and deleting routes.

There are specific commands for each object type.

If no command is given for an object, the current settings are displayed according to the show and list commands (i.e., these are the assumed default commands for an object when no explicit command is given). Other common commands are set for link objects, as well as add and delete for address and route objects. Most commands require additional parameters, e.g., if you want to assign an IP address using ip address add, you will have to specify the address as a parameter.

ip help displays all possible objects, while ip link help shows all commands/parameters pertaining to link objects, including their syntax. The syntax for ip is not always straightforward.

For example, if you want to assign an IP address to a network interface, you might use the following command:

# ip address add local 192.168.2.4/24 device eth0

The local parameter is used to specify that an IP address for a local interface is being set. However, local is the default parameter for ip address add, so it may be left off. ip requires the netmask to be present (e.g., /24).

You can find out more about default parameters by running man 8 ip.

After making IP address assignments with ip, you still need to activate the interface in question, which can be done via # ip link set up:

$ ip address show device eth0
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo-fast qlen 100
  link/ether 00:a0:24:56:e3:72 brd ff:ff:ff:ff:ff:ff
  inet 192.168.2.4/24 brd 192.168.2.255 scope global eth0
# ip link set up device eth0
$ ip address show device eth0
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo-fast qlen 100
  link/ether 00:a0:24:56:e3:72 brd ff:ff:ff:ff:ff:ff
  inet 192.168.2.4/24 brd 192.168.2.255 scope global eth0
  inet6 fe80::2a0:24ff:fe56:e372/64 scope link

Notice the change from <BROADCAST,MULTICAST> to <BROADCAST,MULTICAST,UP> for the eth0 interface.

You can assign network interface aliases using ip:

# ip address add 192.168.2.6/24 device eth0 label eth0:0

The command above establishes eth0:0 as an alias for the eth0 device from the prior example. Now, both 192.168.2.4 and 192.168.2.6 are mapped to the eth0 network interface.

Setting and deleting routes with ip is relatively straightforward:

# ip route add 192.168.2.1/24 via 192.168.0.254
# ip route delete 192.168.2.1/24 via 192.168.0.254

Remember that changes made with the ip command are not persistent and will be lost upon system reboot.

ip Options

Helpful ip options include:

-a, -all

Executes specified command over all objects (command must support this option).

-br, -brief

Print only basic information in a tabular format for better readability. Currently, this option is only supported for the ip address show and ip link show commands.

-d, -details

Output more detailed information.

-h, -human, -human-readable

Output statistics with human readable values, followed by suffix.

-f ex_family, -family ex_family

Specify the protocol family to use. This option accepts a protocol family as an argument (inet, inet6, bridge, decnet, ipx, mpls, link).

-o, -oneline

Output each record on a single line, replacing line feeds with the \ character. This is useful to count records with the wc command or to grep the output.

-r, -resolve

Use the system's name resolver to print DNS names instead of host addresses.

-s, -stats, -statistics

Output more information. If the option appears twice or more, the amount of information displayed increases.

ip Objects

ip supports the following objects:

address

Protocol (IPv4 or IPv6) address on a device.

addrlabel

Label configuration for protocol address selection.

l2tp

Tunnel Ethernet over IP (L2TPv3).

link

Network device.

macsec

802.1AE, the IEEE MAC Security standard.

maddress

Multicast address.

monitor

Watch for netlink messages.

mroute

Multicast routing cache entry.

mrule

Rule in multicast routing policy database.

neigh

Manage ARP or NDISC cache entries.

netns

Manage network namespaces.

ntable

Manage the neighbor cache's operation.

route

Routing table entry.

rule

Rule in routing policy database.

tcp_metrics

Manage TCP metrics.

token

Manage tokenized interface identifiers.

tunnel

Tunnel over IP.

tuntap

Manage TUN (Tunnel)/TAP (Terminal Access Point) devices.

xfrm

Manage IPSec policies.

The names of all objects may be written in full or abbreviated form, e.g., address can be abbreviated as addr or just a.

ip Commands

It is possible to show (or list), add, and delete objects, but some objects do not allow all of these operations or have some additional commands.

ip link show

List network devices (includes MAC addresses).

ip address show

List network devices and include IP addresses. Adding the -s (-stats, -statistics) option makes the output of this command more similar to the output of the older ifconfig command.

# ip link set device ex_interface up

Enable a network interface.

# ip link set device ex_interface down

Disable a network interface.

# ip address add ex_ip_address/ex_netmask device ex_interface

Set IP address for a specific interface.

# ip address delete ex_ip_address/ex_netmask device ex_interface

Delete IP address for a specific interface.

ip route show

Display routing table. Also displays the network's default gateway and its associated IP address.

# ip route add default via ex_gateway_ip_address

Add a default gateway.

# ip route add default via ex_gateway_ip_address device ex_interface

Add a default route for all addresses via a gateway that can be reached on a specific interface.

# ip route add ex_network_address/ex_netmask via ex_gateway_ip_address

Add a static route to a network via a gateway.

# ip route add ex_network_address/ex_netmask via ex_gateway_ip_address device ex_interface

Add a static route to a network via a gateway that can be reached on a specific interface.

# ip route delete ex_network_address/ex_netmask via ex_gateway_ip_address

Delete a static route to a network via a gateway.

# ip route delete ex_network_address/ex_netmask via ex_gateway_ip_address device ex_interface

Delete a static route to a network via a gateway that can be reached on a specific interface.

Internet Protocol Version 6 (IPv6)

With Internet Protocol Version 6 (IPv6), Internet Service Providers (ISPs) assign you a complete network (i.e., a network prefix) that fixes only 48 or 56 of the possible 128 address bits in an IPv6 address. Using IPv6, a computer can assign itself an address that is suitable to communicate with other local computers. Also, without DHCP, using IPv6, a computer can locate routers in the neighborhood that are prepared to forward data to the Internet.

IPv6 datagrams enable more efficient routing. IPv6 defines methods to change a network's network prefix more easily than was previously done. IPv6 supports encryption networks (e.g., IPsec) and mobility where computers can migrate from one network to another without changing addresses or interrupting exiting connections (i.e., mobile IPv6).

The most important properties of IPv6 include:

• The length of addresses is increased from 32 to 128 bits, resulting in a total of 3.4 x 10^38 addresses. The large address space enables more flexible address assignment for various purposes, as well as simplified routing.
• IPv6 stations can automatically obtain configuration parameters from a router when they are connected to a network. If necessary, there is still a DHCPv6 protocol.
• There are only seven fields in an IP header, so routers can process IP datagrams more quickly. If necessary, you can use several headers.
• Extended support for options and extensions, which also contributes to router processing speed.
• Improved transmission of audio and video data, and better support for real-time applications.
• Increased security by means of secured data transmission, and mechanisms for authentication and integrity protection.
• Extensibility to ensure the future of the protocol. IPv6 does not try to cover all possibilities. Instead, the protocol is open to the integration of additional functionality in a backwards-compatible manner.

IPv6 is a targeted replacement for IPv4. Most IP-based protocols, starting with TCP and UDP, remain unchanged. Only at the infrastructure level will some protocols become extraneous or be replaced by IPv6-based versions.

IPv6 addresses are not notated as decimal numbers, but as hexadecimal (base-16) numbers. Four hexadecimal digits are grouped and these groups are separated by colons.

fe80:0000:0000:0000:025a:b6ff:fe9c:406a

Leading zeroes in a group may be omitted and (at most) one run of zero blocks may be replaced by two colons. An abbreviated rendition of the address above could be:

fe80::25a:b6ff:fe9c:406a

The IPv6 address ::1 is an abbreviation of 0000:0000:0000:0000:0000:0000:0000:0001, which corresponds to the IPv4 loopback address, 127.0.0.1. IPv6 does not support broadcast addresses.

IPv6 addresses may be divided into a 64-bit network part and a 64-bit station part. This implies that every IPv6 subnet contains 2^64 addresses, i.e., 2^32 times as many as the whole IPv4 Internet.

Subnetting using variable prefix lengths, as used in IPv4, is not supposed to be a part of IPv6. However, it is assumed that your ISP will provide you with a /56 address prefix, so that you can use 256 subnets with 2^64 addresses each.

You can specify network prefixes by appending a slash and the decimal length to an address. An address like fe80::/16 describes the network where addresses start with fe80 and then continue arbitrarily.

There are three basic types of IPv6 addresses:

1. Unicast addresses apply to one particular network interface (a station may be equipped with several network interfaces, which will each have their own addresses).
2. Anycast addresses refer to a group of network interfaces. These typically belong to different stations and the closet station is supposed to answer (e.g., you may address all routers in an IPv6 network by using the address resulting from appending an all-zero station part to the 64-bit address prefix of the network).
3. Multicast addresses are used to deliver the same packets to several network interfaces. IPv6 does not use broadcast, which is a special case of multicast. For example, the address ff02::1 refers to all stations on the local network.

Also, IPv6 distinguishes various scopes:

• Global scope applies to addresses that are routed within the whole (IPv6) Internet.
• Link-local scope applies to addresses that are not routed and are only valid within the same network. Such addresses are commonly used for internal administrative purposes. Link-local addresses are always located within the fe80::/64 network. The other 64 bits are, in the most straightforward instance, derived from the MAC address of the interface.
• Site-local scope applies to addresses that are only routed within one site. In reality, site-local addresses have been deprecated. Site-local addresses use the fec0::/10 prefix.

• Unique-local addresses are similar to site-local addresses and correspond to the RFC1918 addresses (e.g., 192.168.x.y) of IPv4. However, IPv6 does make it easy to use proper (i.e., globally visible) addresses, so you do not have to resort to using unique-local addresses in order to assign your stations any addresses at all.

  Therefore, there is no compelling reason to use unique-local addresses other than as a fallback position if something is wrong with your real prefix. Unique-local addresses use the fc00::/7 prefix. If set, the first bit following the prefix indicates that the address is locally assigned. This splits the address block into two equally sized blocks, fc00::/8 and fd00::/8.

It is important to emphasize that with IPv6, every network interface can have several addresses. For example, a network interface can get an automatic link-local address, but can have several unique-local or global addresses on top of that with no issues whatsoever. All of these addresses carry equal weight.

A useful command for an IPv6 administrator is ipv6calc, which makes handling IPv6 addresses easier (this command may need to be installed from your distribution's repository, e.g., # sudo apt install ipv6calc):

$ ipv6calc --showinfo fe80::224:feff:fee4:1aa1
no input type specified, try autodetection...found type: ipv6addr
no output type specified, try autodetection...found type: ipv6addr
Address type: unicast, link-local, iid, iid-global, iid-eui48
Registry for address: reserved(RFC4291#2.5.6)
Interface identifier: 0224:feff:fee4:1aa1
EUI-48/MAC address: 00:24:fe:e4:1a:a1
MAC is a global unique one
MAC is an unicast one
OUI is: AVM GmbH
Built-In database: IEEE:OUI/20170625

Above, the address in question is a link-local unicast address whose station part hints at a device manufactured by AVM GmbH.

ipv6calc also serves to convert addresses from one format into another, e.g., you might simulate the method used to derive the station part of an IPv6 address (also called EUI-64) from a MAC address:

$ ipv6calc --in mac --out eui64 00:24:fe:e4:1a:a1
No action type specified, try autodetection...found type: geneui64
224:feff:fee4:1aa1

Ideally, integrating your computer into an IPv6 network would not require any action on your part. The mechanism of Stateless Address Autoconfiguration (SLAAC) makes it possible for everything to automatically take place. With IPv6, SLAAC approximately plays the role that DHCP would play in IPv4, at least for simple applications.

If a new IPv6 network interface is activated, the station first generates the appropriate link-local address. This assumes the fe80::/64 prefix and derives the station part from the MAC address of the interface in question. After that, the station sends a link-local router solicitation (RS) on that interface to the multicast address, ff02::2, which refers to all routers in the subnet.

This causes the router (or routers) on the physical network of the interface to emit router advertisements (RA) containing the prefixes they are routing. On that basis, the station constructs additional (possibly globally visible) addresses for the interface.

RS and RA are part of the Neighbor Discovery Protocol (NDP), which in turn belongs to ICMPv6, the IPv6 counterpart to ICMP. RAs and the IPv6 addresses derived from them only remain valid for a certain time if they are not refreshed. Hence, routers send unsolicited RAs every so often. The RS only serves to avoid having to wait for the next unsolicited RA when a new interface is brought up and makes it possible to obtain the necessary information at once.

The advantage of this approach is that it does not require explicit configuration within a DHCP server. It is also straightforward to obtain redundancy by configuring several routers within the same subnet.

In addition, routers do not need to remember (as they should with DHCP) which station is currently using which IP address (hence, stateless). This does not mean that IPv6 can do away with DHCP altogether (there is DHCPv6), since there are important bits of information that cannot be obtained via SLAAC (e.g., a DNS server).

$ ip address show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
    qdisc pfifo_fast state UP qlen 1000
  link/ether 70:5a:b6:9c:40:6a brd ff:ff:ff:ff:ff:ff
  inet 192.168.178.130/24 brd 192.168.178.255 scope global eth0
  inet6 2001:db8:56ee:0:725a:b6ff:fe9c:406a/64 scope global dynamic
    valid_lft 6696sec preferred_lft 3096sec
  inet6 fe80::725a:b6ff:fe9c:406a/64 scope link
    valid_lft forever preferred_lft forever

The output above contains both the link-local address (scope link, starting with fe80::) and a globally visible address (scope global dynamic, beginning with 2001:), which the interface has obtained via SLAAC. You can also correlate the MAC address (in the link/ether line) with the station parts of the IPv6 addresses.

The station parts of your IPv6 addresses, which are derived from your MAC addresses, are a potential problem for your privacy. If you always use the same source address, it is trivial to correlate your activities with that address.

One way of ameliorating this problem is privacy extensions, which add a random, otherwise unused, station part for outgoing traffic and pick a new one every so often. The privacy extensions can be activated for an interface (here eth0) using the sysctl command and its -w (--write) option:

# sysctl -w net.ipv6.conf.eth0.use_tempaddr=2
# ip link set device eth0 down
# ip link set device eth0 up

To make this setting permanent, enter it into the /etc/sysctl.conf file.

It is still possible to manually assign an IP address using the ip command. For example:

# ip address add 2001:0001:0001:1443::0411/122 device eno55

There are IPv6 versions of many common networking commands, as well, such as:

• ping6
• traceroute6
• tracepath6

Internet Control Message Protocol (ICMP)

Internet Control Message Protocol (ICMP) is used for network management and to report network problems (e.g., a failed connection or an unreachable subnet). The ping program uses two special ICMP messages, ECHO_REQUEST and ECHO_REPLY.

The ICMP packet is encapsulated as data inside of an IP datagram and contains further data fields depending on the code.

Transmission Control Protocol (TCP)

Transmission Control Protocol (TCP) is a reliable, connection-oriented protocol. Unlike the connection-less IP, TCP supports operations to create and bring down connections, which arrange for a virtual connection between the source and destination nodes. Since TCP data is transmitted via IP, the actual data transmission still happens unreliably and on a connection-less basis.

TCP achieves reliability by means of the destination node acknowledging the receipt of each TCP segment (technically, TCP acknowledgments are based on octets, i.e., bytes, rather than segments). Each of the two communicating nodes annotates its segments with sequence numbers, which the other node declares received in one of its next segments. If there is no such acknowledgment within a certain defined period of time, the sending node retries sending the segment in order to try and receive an acknowledgment then.

To avoid a loss of performance, a sliding window protocol is used so that a number of segments can remain unacknowledged at the same time. In spite of this, TCP is considerably slower than IP.

Every TCP segment contains a header of at least 20 bytes in addition to the IP header (remember that the TCP segment including the TCP header is considered data by IP). Errors in the data can be detected based on the checksum.

Every GNU/Linux system supports many independent simultaneous TCP connections distinguished by port numbers. The combination of an IP address and a port number, together with the IP address and port number of a peer, is called a socket. The same TCP port on a node may take part in several TCP connections to different peers, defined by the peer's IP address and port number.

The TCP virtual connection is built using the three-way handshake. Using this three-way handshake, the communication peers agree on the sequence numbers to be used. Two flags in the TCP header, SYN and ACK, play an important role in this.

The first data segment sent to the recipient has the SYN flag set and the ACK flag cleared. Such a segment indicates a connection request. The recipient acknowledges this using a TCP segment that has both the SYN and ACK flags set. The sender, in turn, acknowledges this segment using a segment that has the ACK flag set, but not the SYN flag. At this point, the connection has been established. Subsequent TCP segments also only have the ACK flag set.

At the end of the communication, the connection is torn down by means of two two-way handshakes using the FIN flag.

Two nodes need to agree about the start of a connection, but a connection can be unilaterally torn down. After a unilateral tear-down, data is still passed from the nodes in both directions, since the node that tore down the connection must still acknowledge the data it receives via the remaining part of the connection. However, it can no longer send payload data across the connection, i.e., whichever node torn down the connection can still receive payload data, but not send it.

TCP States

LISTEN

The socket listens for incoming connections. This is the state in which you can find the listening socket of a local TCP server.

SYN-SENT

The socket tries to establish a connection and has sent the first packet of the three-way handshake, but not yet received a reply.

SYN-RECEIVED

This endpoint has received a connection request and sent an acknowledgment. This endpoint is waiting for final acknowledgment that the other endpoint did receive this endpoint's acknowledgment of the original connection request. Results after step two of the three-way handshake.

ESTABLISHED

Represents a fully established connection. This is the normal state for the data transfer phase of the connection.

FIN-WAIT-1

Waiting for an acknowledgment of the connection termination request or for a simultaneous connection termination request from the remote TCP. Normally, this state is of short duration.

CLOSE-WAIT

This endpoint has received a close request from the remote endpoint and this TCP is now waiting for a connection termination request from the local application.

CLOSING

Waiting for a connection termination request acknowledgment from the remote TCP. This state is entered when this endpoint receives a close request from the local application, sends a termination request to the remote endpoint, and receives a termination request before it receives the acknowledgment from the remote endpoint.

FIN-WAIT-2

Waiting for a connection termination request from the remote TCP after this endpoint has sent its connection termination request. Normally, this state is of short duration, but if the remote socket endpoint does not close its socket shortly after it has received information that this socket endpoint closed the connection, then it might last for some time.

Excessive FIN-WAIT-2 states can indicate an error in the coding of the remote application.

LAST-ACK

Waiting for an acknowledgment of the connection termination request previously sent to the remote TCP. This state is entered when this endpoint received a termination request before it sent its termination request.

TIME-WAIT

Waiting for enough time to pass to be sure that the remote TCP received the acknowledgment of its connection termination request.

CLOSED

The socket is not being used. Represents no connection state at all.

User Datagram Protocol (UDP)

Unlike TCP, the User Datagram Protocol (UDP) is a connection-less and unreliable protocol. It is not much more than IP with ports (UDP and TCP may simultaneously use the same port number for different purposes). UDP requires neither the connection initialization of TCP, nor the acknowledgments. Therefore, the protocol is much faster. However, like with IP, data can get lost or jumbled.

UDP is used when there is little data to transmit, so that the cost of a TCP connection initialization is high in comparison, or where not every single bit counts, but delays are unacceptable. For example, with Internet telephony or video transmission, lost datagrams call attention to themselves through cracking noises or snow in the picture. A longer interruption, like the ones common with TCP, would be more noticeable.

Ports

A port is a numeric value assigned to a service. It represents an external endpoint for a logical connection for a node for upper-layer protocols. Ports are provided by both the TCP and UDP protocols at the Transport Layer, and allow a system to maintain more than one connection at a time.

Ports allow a single host with a single IP address to provide multiple network services. There are 65,536 ports each for TCP and UDP. Port number 0 is a signal to the system's TCP/IP stack to pick an otherwise unused port.

TCP/IP ports are broken up into defined ranges:

0-1023

Well-known ports (also known as privileged ports) may only be opened by the root user and are assigned to commonly used network services (i.e., legacy services).

1024-49151

Registered ports are assigned by the Internet Assigned Numbers Authority (IANA) to provide standardization of ports.

49152-65535

Dynamic ports (also known as private or ephemeral ports) cannot be reserved and can be used for any purpose.

These are some of the most important ports and their associated services:

LDAPS, IMAPS, and POP3S are services that may also be accessed via connections that are first established in the clear (i.e., insecurely), and then upgraded to authenticated and encrypted connections later on.

While protocols like TCP and UDP are concerned with moving data from one node to another, services usually rely on application protocols that assign meaning to the data exchanged using the communication protocol.

For example, if you send an email message using SMTP, your computer contacts the remote SMTP server (via TCP on port 25), identifies itself, sends your address, as well as that of the recipient (or recipients) and the actual message, in each case after the remote server prompts for them. The details of this conversation are specified by the application protocol, SMTP.

Dynamic Host Configuration Protocol (DHCP)

Dynamic Host Configuration Protocol (DHCP) is used to save a system administrator from having to define network parameters on every single host in the network. Instead, a GNU/Linux system fetches its network parameters (typically the address of a default router, and one or more DNS servers) from a remote DHCP server when the network adapter is enabled.

DHCP Commands

Most GNU/Linux distributions make it easy to use DHCP for configuration. These are some of the commands that can be used to interact with DHCP on a GNU/Linux system:

# dhclient -r

Release DHCP leases for all network interfaces.

# dhclient -r ex_device

Release DHCP lease for a specific network interface.

# dhclient

Obtain new DHCP leases for all network interfaces.

# dhclient ex_device

Obtain new DHCP lease for a specific network interface.

echo 1 > /proc/sys/net/ipv4/ip_forward

Enable IP forwarding for IPv4.

To make this change persist between system reboots, ensure that the net.ipv4.ip_forward=1 line is present in the /etc/sysctl.conf file and reboot the system after you edit /etc/sysctl.conf to activate the change.

Current DHCP leases should be released before establishing new DHCP leases.

Name Resolution

There are several ways for GNU/Linux systems to resolve hostnames to IP addresses, and vice versa.

Domain Name System (DNS)

The Domain Name System (DNS) is one of the fundamental technologies enabling the scalability of the Internet. Its job is to assign human-readable names to network nodes and to find the corresponding IP addresses (or vice versa). It does this by means of a worldwide distributed database of DNS servers (also referred to as name servers).

/etc/resolv.conf

Usually, programs on a GNU/Linux system do not directly talk to the DNS, but utilize the services of a resolver. Often, this is part of the C runtime library.

The central configuration file for the resolver is /etc/resolv.conf. Usually, GNU/Linux systems have network daemons that automatically manage this file. If you wish to manually edit /etc/resolv.conf, consult your GNU/Linux distribution's documentation for recommendations.

/etc/resolv.conf is used to define the DNS servers that the resolver consults:

nameserver 192.168.10.1
nameserver 192.168.0.99
search foo.example.com bar.example.com example.com

There are five main directives:

1. domain ex_domain The local domain. This is the domain name that the resolver tries to append to incomplete names (typically, names that do not contain a period).

2. search ex_domain_1 ex_domain_2 A search list.

   As an alternative to a single entry using domain, you can specify a list of several domain names to be appended to incomplete names. The entries in the list are separated by spaces. At first, the resolver tries the unchanged name. If that fails, the list entries are appended in order and these names are tried.

   domain and search are mutually exclusive. If both occur in /etc/resolv.conf, whichever line is last in the file is used.

3. nameserver ex_ip_address The local DNS server. The local resolver will consult the DNS server given here. You may define up to three name servers in separate nameserver directives, which will be consulted in sequence, if required.

4. sortlist ex_ip_address/ex_netmask The sort order. If several IP addresses are returned for a name, the one matching the specification here will be preferred. In the sort list, there is room for up to ten entries.
5. options ex_option Options. This is used for specific resolver settings, which are denoted (along with their default values). In practice, these are seldom, if ever, changed.

These are the options available for the options directive in /etc/resolv.conf:

debug

Regular log messages are output to the standard output (commonly unimplemented).

ndots ex_number

The minimum number of dots within a name that will cause the resolver to perform a direct query without accessing the search list.

attempts ex_number

The number of times the resolver will query a server before giving up. The maximum value is 5.

timeout ex_number

The initial time out for query attempts, in seconds. The maximum value is 30.

rotate

Not only the first, but all specified servers will be queried in rotation.

no-check-names

Deactivates the standard check that determines whether returned hostnames only contain allowable characters.

/etc/hosts

An alternative to DNS is the local resolution of hostnames and IP addresses by means of the /etc/hosts file. Often, this is only of interest for small networks that are not connected to the Internet. If you only need to deal with several computers, it can be simpler to simply configure the DNS on the clients' side, and assign names and addresses to your own computers using /etc/hosts.

The syntax for the /etc/hosts file is:

ex_ip_address ex_hostname ex_alias

An typical /etc/hosts file may look like this:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

The content of the /etc/hosts file is plain ASCII text that may contain line-based entries, as well as comments starting with #. These entries contain an IP address in the first column and the fully qualified domain name (FQDN) of a host in the second column.

Also, it is permissible to add more names on the same line. Spaces or tabs can be used to separate columns.

Name Resolution Implementation

The exact mechanisms the C library uses for name resolution are controlled by the /etc/nsswitch.conf file. For example, /etc/nsswitch.conf determines which name resolution services are used and in which order.

In addition, there are rules for the resolution of usernames and groups. Run man 5 nsswitch.conf for a detailed description of its syntax and function.

The part of /etc/nsswitch.conf relevant to hostname resolution looks like this:

hosts: files dns

files means to refer to the local files (i.e., /etc/hosts) for name resolution. If there is no entry there, a DNS lookup is next and /etc/resolv.conf will be referenced for name servers to utilize for the DNS lookup.

If an entry is found for a hostname in /etc/hosts, even if it is incorrect, no further lookups will occur.

Other name resolution-related files are:

• /etc/hostname Used to statically store the hostname of a system.
• /etc/networks Used to map a network name to an otherwise IP-related network (i.e., a network name with a network address).
• /etc/host.conf Sets the order in which name resolution sources are searched. This file is overridden by /etc/nsswitch.conf.

Name Resolution Commands

# systemctl restart NetworkManager.service

Restart the Network Manager service and flush the DNS cache.

host ex_hostname

Resolve hostname to IP address using only DNS.

By default, the host command uses the name servers specified in /etc/resolve.conf, but an optional name server hostname or IP address may be appended to the command.

host ex_ip_address

Resolve an IP address to a hostname using only DNS.

host -l ex_zone_name

Perform a zone transfer and print out the NS, PTR, and A/AAAA records for the zone name (if you have the appropriate permissions).

host -al ex_zone_name

Print out all DNS records in the zone.

dig ex_hostname

Resolve hostname to IP address using only DNS. A name server and DNS record type can be specified with the command (e.g., dig @ex_name_server ex_hostname ex_dns_record_type).

@ex_name_server can be the name server's hostname or IP address and ex_dns_record_type can be a valid DNS record type (e.g., MX, NS, A, SIG, ANY). If you do not specify a DNS record type, dig does an A record query by default.

The output of a dig query is structured:

HEADER

Contains information about the dig environment and options.

QUESTION

Echoes back your query.

ANSWER

The reply to your query.

AUTHORITY

The servers that are the authoritative name servers for the requested target. A name server is considered to be authoritative if it has a record for the domain name being requested in its database of name mappings.

ADDITIONAL

A catch-all section, usually displaying the name servers for the target.

STATISTICS

How much time it took in milliseconds or seconds to answer the query, your system's name server, as well as the date and time of the query.

If you want abbreviated output from dig, add the +nostats +nocomments +nocmd options. For the most terse output, use the +short option.

dig -x ex_ip_address

Resolve an IP address to a hostname using only DNS.

getent hosts ex_hostname

Resolve a hostname to an IP address using both /etc/hosts and DNS.

getent utilizes a system's /etc/nsswitch.conf file and can perform IP translation on Network Information Service (NIS) (formerly known as Yellow Pages) and LDAP servers, as well.

hostname

Display system hostname.

The hostname command is linked to the following related commands:

• domainname
• dnsdomainname
• nisdomainname
• ypdomainname

hostname -f, hostname --fqdn, hostname --long

Display system's Fully Qualified Domain Name (FQDN).

hostname -I, hostname --all-ip-addresses

Display system's local IP addresses.

Extended Internet Service Daemon (xinetd)

The Extended Internet Service Daemon (xinetd) is an open source super-server daemon. It offers a more secure alternative to the older Internet Daemon (inetd). xinitd is used to listen in the place of many services and start the service that handles the incoming request only when it arrives on the system.

Keep in mind, any time you put services behind xinetd, they must be started to function and that means a small, but noticeable, delay.

Services that may be managed by xinetd include:

• chargen
• daytime
• echo
• ftp
• pop3
• rsync
• smtp
• telnet
• tftp (Trivial File Transfer Protocol)
• time
• vnc

xinetd brings together many features (e.g., port supervision, access control, and logging) and uses a single, centralized configuration file, usually /etc/xinetd.conf. In this file, blank lines or lines beginning with a # character are ignored.

The configuration settings are collected in sections. Each section begins with a keyword corresponding to a service name from /etc/services and contains assignments of values to attributes like:

default
{
    ex_attribute ex_operator ex_parameter...
}
service ex_service_name
{
    ex_attribute ex_operator ex_parameter...
}

In practice, the most important operator is =, which assigns a specific set of values to an attribute. Attributes that can have several values at once also support the += and -= operators to add or remove values, respectively.

The default section contains default settings that apply to all services, as long as their sections do not contain more specific values. If they do, the default values are either replaced by or combined with the more specific ones. Each additional section gives more detailed information for a specific service.

Important attributes include:

access_times

The service is only available at the given times.

cps

Limits the number of connections per second allowed through xinetd. Any more connections than the maximum causes the service to be temporarily disabled before xinetd can check again to see whether it can accept more connections.

disable

Deactivates the service.

disabled

Can be set in the defaults section to disable a number of services, e.g., disabled finger ftp.

interface

IP address for the interface used by xinetd to listen for service requests.

instances

Maximum number of simultaneous instances of the service (if wait = no).

log_on_failure

Defines what information to log on an unsuccessful connection attempt, e.g., ATTEMPT (the failed attempt).

log_on_success

Determines what information to log on a successful connection attempt, e.g., HOST (the client's name), USERID (the client user according to RFC1413).

log_type

Determines the type of logging done by xinetd, e.g., SYSLOG or FILE.

no_access

These clients may not access the service.

only_from

Only the specified clients (as DNS names, IP addresses, or network addresses) may access the service.

protocol

The protocol used by the service. Must be in /etc/protocols.

server

Filename of the actual server program.

server_args

Invocation parameters for the server program.

socket_type

Includes values like stream, dgram, or raw.

type

Allows specifications like INTERNAL (i.e., the service is directly implemented by xinetd) or UNLISTED (i.e., the service has no entry in /etc/services).

user

The user whose privileges are used to execute the service. This must be a valid system user.

wait

If yes, it is a single-threaded service. no allows xinetd to simultaneously start the service several times.

The set of allowable attributes in a section depends on the service to be configured. The exact syntax, as well as more attributes, can be found in the documentation for the xinetd.conf file.

xinetd also has an include directory, /etc/xinetd.d/. This directory includes a single text file for each governed service and any special settings that need to be set for correct functioning. Each file is exactly named as the service that appears in the /etc/services file.

xinetd can be passed a number of options to control the way it works:

-f ex_file

Causes xinetd to read ex_file as its configuration file. The default value is /etc/xinetd.conf.

-filelog ex_file

Determines the filename used by xinetd to write its log messages to. This option and the -syslog option are mutually exclusive.

-inetd_compat

Causes xinetd to read inetd's configuration file, /etc/inetd.conf, in addition to its own configuration file.

-syslog ex_category

Causes xinetd to send its log messages to the syslog daemon within the given category. Possible categories include daemon, auth, user, and the eight categories local0 to local7.

After changing /etc/xinetd.conf or a value in any of the files in /etc/xinetd.d/, you need to restart the xinetd daemon:

# service xinetd restart

Services that are managed by xinetd can be started and stopped with the service command:

# service ex_service start

Start a service managed by xinetd.

# service ex_service stop

Stop a service managed by xinetd.

It is also possible to control xinetd by means of signals. The most important signals and their effects are:

SIGHUP

Causes a hard reconfiguration. xinetd rereads its configuration file and terminates the servers that are no longer enabled in the new configuration. For all other servers, access control is redone and all connections that do not pass are reset.

If there are more connections to a service than allowed by instances, a random set of servers is terminated until the limit is observed again.

SIGIOT

Causes xinetd to perform an internal consistency check in order to ensure that its data structures have not been damaged.

SIGQUIT

Terminates xinetd.

SIGTERM

Terminates all active services, then xinetd.

SIGUSR1

Creates a memory dump in file /var/run/xinetd.dump.

Services started by xinetd can be subdivided into two groups (see the wait and instances attributes):

1. If a new process is started for each access to the service, the service is called multithreaded.
2. If a service only accepts a new request after the preceding one has finished, the service is called single-threaded.

Datagram-based services (i.e., those based on UDP) are frequently single-threaded, while TCP-based services are always multi-threaded.

xinetd Example

To define a service that produces a process list when the hypothetical ps port is accessed and limit access to this service to the local host, you can define the following:

service ps
{
    socket_type = stream
    protocol = tcp
    wait = no
    user = root
    server = /bin/ps
    server_args = auxw
    only_from = localhost
    no_access =
}

You can configure xinetd such that the ps service is only available on the loopback interface (127.0.0.1) by utilizing the interface attribute.

Limiting a service to the local host by binding it to the loopback address has the advantage that access is restricted to local processes. You will need to ensure that local processes do try to contact localhost instead of the host's FQDN.

TCP Wrappers

If a GNU/Linux service references the libwrap.a library, a TCP wrapper can be used to wrap or protect it, without xinetd.

When used with xinetd, the concept of using TCP wrappers is to place a controlling daemon with instructions as to who is allowed and denied access to a service in front of each service that runs on a machine. The services are configured to run as parameters of the tcpd daemon, which references the /etc/hosts.allow and /etc/hosts.deny files.

To use TCP Wrappers, you first need to enable the functionality in each service’s configuration file in /etc/xinetd.d/. Follow these steps:

1. Verify that the tcpd package is installed.
2. Open the service's file in the /etc/xinetd.d/ directory.
3. Comment out the existing server = line.
4. Add the following line, server = /usr/sbin/tcpd.
5. Add the following line, server_args = ex_path_to_service_daemon.
6. Save the file and restart xinetd.

Next, you need to create access controls. tcpd uses the /etc/hosts.allow and /etc/hosts.deny files to specify who can access the services it manages. Hosts in /etc/hosts.allow are allowed access, while those in /etc/hosts.deny are not.

The format of a line in /etc/hosts.allow and /etc/hosts.deny is:

ex_daemons: ex_hosts : ex_option : ex_option

In the two option fields, you can use the twist keyword to booby-trap what would appear to be a service with another program or action (e.g., in.fingerd: ALL : twist '/bin/echo' 'Go away!').

You can use the spawn keyword to allow a host to connect to a service, but at the same time be notified of the access (e.g., in.telnetd: All : spawn '/bin/echo' 'date' from %h | mail root). If you spawn a command, it runs as a child of the current shell, but if you use twist to run the command, it replaces the current process with the command that you run.

Daemons can be specified via several methods, including:

@group

Used by the Network Information Service (NIS) to denote a Net group.

ex_network_address/ex_netmask

ex_netmask can use either the dotted octet or /xx notation.

All

Means all services.

ex_daemon,ex_daemon

Multiple daemons affected by an entry should be separated by spaces or commas.

Hosts can be specified by many methods, as well:

hostname

Affects a single unqualified hostname, typically from the local domain.

hostname.example.com

Affects a resolvable, fully qualified hostname, typically from a remote domain.

/path/ex_filename

Causes the listed file (full path and filename) to be referred to as the list of hosts that will be affected.

ex_service

A single service name affects only that service.

/etc/hosts.allow is read first. If a match is found, /etc/hosts.deny is ignored. Changes in these files become effective immediately. Likely, it is best to deny all access in /etc/hosts.deny and then to whitelist specific hosts in /etc/hosts.allow.

/etc/hosts.allow and /etc/hosts.deny are only read if they exist. If they do not exist, no rules can be applied, allowing complete access for services, as far as TCP wrappers are concerned.

Networking and systemd

There are persistent services, which are started when the system is booted, and services that are started on-demand by means of socket activation. From systemd's point of view, persistent network services do not greatly differ from other persistent services that are started and managed by systemd.

They take care of their own connections and systemd has nothing to do with them. systemd's only job consists of starting the service at the correct time and potentially keeping an eye on the service, in case it misbehaves.

For a network service, there needs to be a unit file for the service and it should be known to systemd by enabling the service (i.e., # systemctl enable ex_service). After that, systemd should automatically take care of the service, although you can choose to manually manage the service using the systemctl command.

Many services expect to be started when the network is available. However, this is a vague concept. systemd uses three different targets to handle the network is available concept:

1. network-pre.target
2. network.target
3. network-online.target

network-pre.target

network-pre.target is a target that is used by services that need to be started before the network is activated. For the most part, this includes services that configure a firewall, so that it is available before network interfaces are brought up.

This precludes a vulnerable phase between the start of networking and the initialization of the firewall. This target cannot be manually activated. Network management software should depend on network-pre.target, but only in the temporal sense (avoid an explicit dependency using Requires or Wants):

[Unit]
After=network-pre.target

Services that are to be started before the network is activated should have the following in their configuration:

[Unit]
Wants=network-pre.target
Before=network-pre.target

network.target

network.target only stipulates that the local network software was initialized. It does not make any assumptions as to whether actual network interfaces have been configured.

The main purpose of this target is to provide a synchronization point during a shutdown of the system. If all services that presume availability of the network are started after network.target is reached, then that implies that on shutdown they will be deactivated before network.target is deactivated. This means that no network service has its connection forcibly severed.

You may not manually start network.target, and neither is the target presupposed by network services using Requires or Wants. Such a clause should only occur in the configuration of the network management software.

Services that need to be shut down before the network is brought down should have the following in their systemd configuration:

[Unit]
After=network.target

network-online.target

network-online.target explicitly waits for the network to become available, where the exact meaning of becoming available is determined by the network management software (e.g., a configured external IP address would be a common prerequisite). In principle, services that depend on the network can have a Requires or Wants for this target.

Many network services have no problem serving local clients, even though no external network connection is available. network-online.target is more useful for clients that want to access remote services and do not work without a network connection.

Services that are to be started after the network is activated should have the following in their configuration:

[Unit]
Wants=network-online.target
After=network-online.target

Socket Activation

Socket activation is the idea of starting a service only when connection requests to it are received. systemd recommends socket activation for UNIX domain sockets in particular, but there is no problem with using it for TCP or UDP sockets.

There are three basic scenarios where socket activation is worthwhile:

1. When the system is booting, socket activation can enhance parallelism and avoid explicit dependencies. systemd initializes the required communication channels and orchestrates the launch, in parallel, of the corresponding services when requests are received.

   This is useful for services that are used often and permanently, and that should be started as soon as possible (e.g., syslog or D-Bus).

2. Rarely used services are started on-demand, by means of systemd opening the known port associated with the service and listening for connection requests. If connection requests do arrive, systemd launches the actual service and passes the listening socket to the service. Then, the service can handle further requests on its own.

   One example of this is the printer service, CUPS.

3. Rarely used services are started on-demand, by means of systemd opening the known port associated with the service and listening for connection requests. If connection requests do arrive, systemd launches the actual service and passes it the socket for the actual connection. The service handles that specific connection and then terminates again.

   This is less efficient than the other two scenarios, but such services are very convenient to implement, since systemd takes care of all the networking requirements. Services where this approach makes sense include FTP or SSH, especially on hosts where they are infrequently used. This avoids having daemons idly sleeping in the background.

Enable Socket Activation For Secure Shell (SSH) Service Example

To enable socket activation with systemd for a service like the Secure Shell (SSH), we first take a look at how the respective configuration would look like in inetd or xinetd:

ssh stream tcp nowait root /usr/sbin/sshd sshd -i

The -i option ensures that the SSH daemon, sshd, works with inetd, rather than as a free-standing service.

For xinetd:

service ssh {
    socket_type = stream
    protocol = tcp
    wait = no
    user = root
    server = /usr/sbin/sshd
    server_args = -i
}

You also need to know that the Secure Shell uses TCP port 22. In the inetd configuration, this is implicitly denoted by the ssh in the first column. This is a reference to the /etc/services file, which contains the actual port number.

To construct an equivalent configuration for systemd, you need two unit files. First, describe the port that systemd listens on (on behalf of SSH). This is done in a unit file called sshd.socket:

# sshd.socket
[Unit]
Description=Secure Shell service (socket for socket activation)

[Socket]
ListenStream=22
Accept=yes

[Install]
WantedBy=sockets.target

A second unit file is used to start instances of the SSH daemon, if they are required. Since several SSH connections may be active at the same time, we define this unit file as a template, sshd@service:

# sshd@.service
[Unit]
Description=Secure Shell service (per-connection server)

[Service]
ExecStart=-/usr/sbin/sshd -i
StandardInput=socket

In ExecStart, we specify how the daemon is to be started (the minus sign, -, at the start of /usr/sbin/sshd declares that exit statuses other than zero should be considered successful). StandardInput=socket ensures that the daemon can actually talk to the client side (standard output and standard error content go along with this).

By using two unit files to implement this solution, systemd provides a flexible approach. Specifically, the listening port and the actual connections are cleanly separated from one another. For example, one could stop the responsible unit for the port (and thereby inhibit further connections) without influencing existing connections.

Using these two files, you can reload systemd and launch the service:

# systemctl daemon-reload
# systemctl start sshd.socket

To enable the service on a permanent basis, run # systemctl enable sshd.socket.

It is wise to configure SSH for systemd in this way. However, many GNU/Linux distributions rely on a SSH daemon that is permanently running in the background:

[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=sshd.service

This example shows how you can create systemd .socket and .service unit files that implement the classic DAYTIME service, which provides the current date and time on TCP port 13. This is the daytime.socket file:

[Unit]
Description=DAYTIME service socket

[Socket]
ListenStream=13
Accept=yes

[Install]
WantedBy=sockets.target

This is the daytime@.service file:

[Unit]
Description=DAYTIME service (per-connection server)

[Service]
ExecStart=-/bin/date
StandardInput=socket

systemd replaces inetd, but not necessarily xinetd, as the latter contains a few built-in conveniences that systemd does not offer. You can probably do without the standard services ECHO, DISCARD, TIME, and DAYTIME, and various service options are no longer relevant today (e.g., TCPMUX).

You might possibly miss TCP Wrappers, although you may be better served by an actual firewall, and nothing prevents you from using the tcpd program with systemd.

Network Configuration

GNU/Linux systems access the network by means of modems, Ethernet, Wireless LAN (WLAN) adapters, or similar devices. Like other hardware, a network interface on a GNU/Linux system is controlled by the kernel, usually by means of modular drivers that are dynamically loaded on-demand (e.g., via /lib/modules/).

Unlike hard disk partitions or printers, network devices do not appear as device files in the /dev/ directory, but are accessed via interfaces. Network interfaces are a connection channel between a device and a network. They are virtual in the sense that the kernel makes them available after a suitable driver has been loaded and that they can be accessed through more than one (mostly) independent interface.

Network interfaces can be a physical Network Interface Card (NIC) or be abstracted in software. A system can have multiple network interfaces operating at once. Network interfaces are named. For example, a typical name for an Ethernet interface is eth0.

Network adapters are recognized by the kernel when the system is booted. It can identify the correct driver by means of the adapter's Peripheral Component Interconnect ID (PCI ID). It is up to the udev infrastructure to name the device and load the driver.

Each GNU/Linux distribution has an interface configuration file (whose settings apply to all interfaces) and an interface configuration directory (whose files are specific to a certain interface; each file in this directory is named with the name of the corresponding network interface).

Traditionally, these files were used to make manual, persistent network-related changes to a system. Now, most GNU/Linux distributions use a network manager to handle this task. Usually, this network manager (e.g., for Debian and Fedora) is NetworkManager.

NetworkManager

NetworkManager is a GNU/Linux network configuration tool suite. It supports a broad range of networking setups, and integrates well with myriad desktop environments and server configuration management tools.

Likely, your GNU/Linux distribution is already configured to use NetworkManager and the NetworkManager.service has been set to run on system startup. You can confirm this by running:

$ systemctl status NetworkManager.service
● NetworkManager.service - Network Manager
   Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-07-30 08:46:55 PDT; 21min ago
     Docs: man:NetworkManager(8)
 Main PID: 491 (NetworkManager)
    Tasks: 4 (limit: 4686)
   Memory: 14.1M
   CGroup: /system.slice/NetworkManager.service
           ├─491 /usr/sbin/NetworkManager --no-daemon
           └─565 /sbin/dhclient -d -q -sf /usr/lib/NetworkManager/nm-dhcp-helper -pf /run/dhclient-enp1s0

NetworkManager Configuration

/etc/NetworkManager/NetworkManager.conf

The main configuration file for NetworkManager is /etc/NetworkManager/NetworkManager.conf, which is used to configure the various aspects of NetworkManager's behavior.

If a default /etc/NetworkManager/NetworkManager.conf file has been provided by your GNU/Linux distribution, it is best not to modify it, as your customizations may be overwritten by package updates.

/usr/lib/NetworkManager/conf.d/, /run/NetworkManager/conf.d/

Packages and scripts may also place configuration snippets into /usr/lib/NetworkManager/conf.d/ and /run/NetworkManager/conf.d/, respectively.

These files are processed before /etc/NetworkManager/NetworkManager.conf (/usr/lib/NetworkManager/conf.d/ first).

/etc/NetworkManager/conf.d/

You can add your own .conf files to the /etc/NetworkManager/conf.d/ directory. Files in this directory will be read, in order, with later files overriding earlier ones.

For detailed information on how to create these files, run man 5 NetworkManager.conf.

After making a configuration change, be sure to reload NetworkManager:

# systemctl reload NetworkManager

NetworkManager offers several configuration interfaces. The primary interface tool is a command line application, nmcli. A graphical user interface tool is also available, nm-connection-editor.

nmcli

nmcli can be used to create, display, edit, delete, activate, and deactivate network connections. Also, it can control and display network device statuses.

The syntax for nmcli is:

nmcli ex_options ex_object ex_command

ex_object can be:

• general
• networking
• radio
• connection
• device
• agent

There are many possible nmcli commands. Run man 1 nmcli and man 7 nmcli-examples for more information.

A few of the most useful nmcli commands are:

nmcli general

Display NetworkManager status and permissions.

nmcli -t -f RUNNING general, nmcli --terse --fields RUNNING general

Determine if NetworkManager is running.

nmcli device show

Display detailed information about all network devices (compare and contrast to the output of ip address show). Add the -p (--pretty) option to make the output even more visually appealing.

nmcli device show ex_id...

Display detailed information about a specific device. ex_id... can be id, uuid, path, or apath.

$ nmcli -p device show enp1s0
===============================================================================
                            Device details (enp1s0)
===============================================================================
GENERAL.DEVICE:                         enp1s0
-------------------------------------------------------------------------------
GENERAL.TYPE:                           ethernet
-------------------------------------------------------------------------------
GENERAL.HWADDR:                         52:54:00:E3:93:61
-------------------------------------------------------------------------------
GENERAL.MTU:                            1500
-------------------------------------------------------------------------------
GENERAL.STATE:                          100 (connected)
-------------------------------------------------------------------------------
GENERAL.CONNECTION:                     Wired connection 1
-------------------------------------------------------------------------------
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/4
-------------------------------------------------------------------------------
WIRED-PROPERTIES.CARRIER:               on
-------------------------------------------------------------------------------
IP4.ADDRESS[1]:                         10.0.2.15/24
IP4.GATEWAY:                            10.0.2.2
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 10.0.2.2, mt = 100
IP4.ROUTE[2]:                           dst = 10.0.2.0/24, nh = 0.0.0.0, mt = 100
IP4.DNS[1]:                             10.0.2.3
-------------------------------------------------------------------------------
IP6.ADDRESS[1]:                         fec0::e9c2:33c4:4685:7db0/64
IP6.ADDRESS[2]:                         fec0::5054:ff:fee3:9361/64
IP6.ADDRESS[3]:                         fe80::5054:ff:fee3:9361/64
IP6.GATEWAY:                            fe80::2
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 100
IP6.ROUTE[2]:                           dst = fec0::/64, nh = ::, mt = 100
IP6.ROUTE[3]:                           dst = ::/0, nh = fe80::2, mt = 100
-------------------------------------------------------------------------------

nmcli device status

Display status of all network devices.

$ nmcli device status
DEVICE  TYPE      STATE      CONNECTION         
enp1s0  ethernet  connected  Wired connection 1
lo      loopback  unmanaged  --

nmcli device connect ex_interface_name

Connect a device.

NetworkManager will try to find a suitable connection that will be activated. If no compatible connection exists, a new profile with default settings will be created and activated.

nmcli device disconnect ex_interface_name

Disconnect a device and prevent the device from automatically activating further connections without user/manual intervention.

nmcli device wifi list

List nearby Wi-Fi networks.

nmcli device wifi connect ex_ssid_bssid password ex_password

Connect to a Wi-Fi network.

nmcli connection show

Display in-memory and on-disk connection profiles (including each connection's UUID).

$ nmcli connection show
NAME                UUID                                  TYPE      DEVICE
Wired connection 1  99369c6f-13fd-4ea6-b2a7-7ba570441b4d  ethernet  enp1s0

nmcli connection show ex_id...

Display in-memory and on-disk connection profile (including a connection's UUID) for a specific connection. ex_id... can be id, uuid, path, or apath.

nmcli connection show --active

Display only active connection profiles.

nmcli -f ex_field_1,ex_field_2 connection show ex_connection, nmcli --fields ex_field_1,ex_field_2 connection show ex_connection...

Display selected field/value pairs for a connection.

$ nmcli -f ip4.address,ipv4.dns connection show 'Wired connection 1'
ipv4.dns:                               --
IP4.ADDRESS[1]:                         10.0.2.15/24

# nmcli connection reload

Reload all connection files from disk.

# nmcli connection add con-name ex_connection_name

Create a new connection using specified properties.

For example, the following command creates a new Ethernet connection and assigns it a static IP address:

# nmcli connection add con-name eth3 type ethernet ifname eth3 \
    ipv4.method manual ipv4.address 10.10.10.6/24 \
    ipv4.gateway 10.10.10.1
Connection 'eth3' (98abc72b-6041-4332-80cd-6af49c681da3) successfully added.

# nmcli connection delete ex_connection...

Delete a configured connection. ex_connection can be id, uuid, or path.

# nmcli connection modify ex_connection

Add, modify, or remove properties in a connection profile. ex_connection can be id, uuid, or path.

For example, the following command changes the Ethernet BOOTPROTO from manual (static) to auto (DHCP) for the eth3-conn connection:

# nmcli connection modify eth3-conn ipv4.method auto

After a connection is modified, make sure to restart the connection:

# nmcli connection up ex_connection

# nmcli connection edit ex_connection

Edit an existing connection or add a new one using an interactive editor. Once at the nmcli command prompt, you can enter help to see the available commands and quit to exit.

ex_connection can be id, uuid, or path.

After a connection is edited, make sure to restart the connection:

# nmcli connection up ex_connection

nmcli connection up ex_connection...

Activate a connection. ex_connection can be id, uuid, or path.

nmcli connection down ex_connection...

Deactivate a connection from a device without preventing the device from further auto-activation. ex_connection can be id, uuid, path, or apath.

nmcli connection monitor ex_connection...

Monitor connection profile activity in real-time. Prints a line whenever the specified connection changes.

ex_connection can be id, uuid, or path.

# nmcli connection modify ex_connection ipv4.gateway 'ex_gateway_ip_address'

Set IPv4 address for a connection's default gateway.

# nmcli connection modify ex_connection +ipv4.routes 'ex_network_address/ex_netmask ex_gateway_ip_address'

Add a static IPv4 route to a network connection via a gateway.

# nmcli connection modify ex_connection -ipv4.routes 'ex_network_address/ex_netmask ex_gateway_ip_address'

Delete a static IPv4 route to a network connection via a gateway.

nmcli radio wifi off

Turn off Wi-Fi.

Documentation

The NetworkManager documentation is comprehensive and covered across numerous manual pages. For more information, run the following commands:

• man 8 NetworkManager
• man 5 NetworkManager.conf
• man 5 nm-settings
• man 1 nmcli
• man 1 nm-connection-editor

Also, the ArchWiki Network configuration and NetworkManager pages are helpful.

Network Troubleshooting

A basic checklist for troubleshooting an issue is as follows:

1. Gather information
2. Identify what has changed
3. Create a hypothesis
4. Determine the appropriate fix
5. Implement the fix
6. Ensure user satisfaction
7. Document the solution

When troubleshooting a network issue, a good first step is to confirm that the network adapter is present and recognized by the system. If applicable, confirm that the network cable is still seated in the correct socket.

By running, ip -a -s link, you can view an overview of all network interfaces inside a system, even ones that are not currently configured. At least lo and eth0 (if the computer is networked using Ethernet) should be visible. If not, there is likely something wrong with the network adapter's driver or the adapter is not being recognized.

If ip -a -s link does not show your network adapter, run the lsmod command to check whether or not the driver module for the network adapter in question was loaded. If you are not sure which driver module is supposed to be used for the network adapter, consult the output of the lspci -k command for a portion that pertains to the network adapter in question (the -k option tells lspci to show the kernel drivers handling each device).

If the output of ip -a -s link shows the network interface and the parameters displayed with it seem reasonable (i.e., appropriate IP address, netmask, broadcast address), it is time for connectivity tests.

Connectivity Tests

ping

The simplest tool for a network connectivity test is the ping command:

ping ex_host

ex_host can be the IP address associated with a host or its hostname. ping tries to send an ICMP ECHO_REQUEST packet to the host in question. The host should reply with an ICMP ECHO_REPLY packet, which ping receives and reports.

First, confirm that your system can communicate with itself by pinging the loopback address:

$ ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.064 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.042 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.045 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.031 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.044 ms
^C
--- 127.0.0.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 93ms
rtt min/avg/max/mdev = 0.031/0.045/0.064/0.011 ms

You can press Ctrl+c to send a SIGINT signal to interrupt the process and view the ping statistics.

The 56(84) in the ping output above is information regarding the IP datagram header. Without options, an IP datagram header is 20 bytes long. Added to that is the header of an ICMP ECHO REQUEST packet at 8 bytes. This explains the difference between 56 and 84 (i.e., 84 - 56 = 28).

The number 56 results from the fact that ping normally ensures that exactly 64 bytes of payload data are transmitted inside each IP datagram, namely the 8-byte ICMP header and 56 bytes of padding. If enough padding is available, namely at least the size of a struct timeval in C (about eight bytes), ping uses the start of the padding for a timestamp to measure the packet round-trip time.

The next step is to ping the network adapter interface. If that is working, the ping output should be comparable to what was seen above. If you are not seeing any errors up to this point, the odds are that the basic networking functionality of the GNU/Linux system is working. Likely, additional sources of trouble are elsewhere in the network or farther up the system's protocol stack.

The next ping test is for the network's default gateway (or another host on the local network). If you encounter a problem here, there could be an issue with how the netmask is configured. Another possibility is a hardware issue with your system's network adapter, at least if the problem is intermittent.

Continue by pinging hosts outside of the local network. An ECHO_RESPONSE greater than 1000 ms indicates that there is likely some congestion between your host and the target host. If you do not get appropriate answers, you may be encountering a routing issue, or possibly a firewall that is interfering with ICMP traffic.

ping supports many options that extend its testing possibilities. Useful options include:

-a

Audible pings.

-b

Allow pinging a broadcast address.

-c ex_count

Number of ECHO_REQUEST packets to be sent (ping will exit afterwards).

-f

Flood ping. A period (.) is output for every ECHO_REQUEST packet sent, and backspace character for every ECHO_REPLY packet received. The result is a row of periods that tells you how many packets have been dropped during transmission (this option is useful for testing intermittent connection issues).

If you have not simultaneously specified an interval with the -i option, ping sets the interval to 0 and outputs packets as fast as they come back, or 100 times per second. This command needs to be run as the root user, as normal users are limited to a minimum interval of 0.2 seconds.

-i ex_interval

Waits the number of seconds passed to the option as an argument between sending each packet. The default is 1 second, except when flood pinging as the root user.

-I ex_interface

Sets the sender address for the packets. The sender is provided to the option as an IP address or the name of an interface (in which case the IP address of that interface will be used).

-n

Display without DNS name resolution, i.e., numeric output only.

-s ex_packet_size

Determine the size of the padding in bytes. The default value is 56.

Sometimes, there are problems with very large packets that must be fragmented and ping can help diagnose these by means of this option.

traceroute and tracepath

When you have confirmed that a networking issue is related to a routing problem, you can use traceroute to gather additional information.

First, use ip route show to confirm that the correct default route is being used. If a ping to the default gateway works, but a ping to a host beyond the gateway does not, there may be something wrong with the default gateway. Check whether or not your host is reachable from the default gateway (keep in mind that the default router may be running a packet filter that blocks ICMP).

Essentially, traceroute is an extended form of ping. It does not merely check a remote host for responsiveness, but displays the route that packets take through the network. It keeps track of the routers the packet passes through and the quality of the connection to the routers in question.

Unlike ping, traceroute is not based on ICMP, but (traditionally) on UDP. traceroute sends three UDP datagrams to arbitrary ports on the destination node. The first three datagrams have a time to live (TTL) of 1, the next three a TTL of 2, and so on. The first router on the way to the destination decrements the TTL by 1.

The first round of datagrams, which only had a TTL of 1, are dropped and the sender gets an ICMP TIME EXCEEDED message, which (being an IP datagram) contains the router's IP address. The second three datagrams are dropped by the second router and so on.

This way, you can follow the exact route of the datagrams towards the destination. The destination node itself does not send a TIME EXCEEDED message, but a PORT UNREACHABLE message, so that traceroute knows that the route tracing is complete.

The traceroute output consists of several numbered lines. One line corresponds to a group of three datagrams. It shows the node sending the TIME EXCEEDED message, as well as the transmission time of the three datagrams.

Any asterisks in the output mean that there was no answer for one of the datagrams within about five seconds. This is an indication that the route is either configured to not return ECHO REQUESTS from the traceroute command, it is too busy, or it is down and cannot respond. Typically, this is the bottleneck you are looking for.

Today, the traditional traceroute technique based on UDP datagrams does not work in all cases, due to aggressive firewalls that drop datagrams addressed to unlikely UDP ports. You can use traceroute's -I (--icmp) option to get traceroute to use ICMP (ICMP ECHO) instead of UDP (which makes it essentially work like ping).

If you need to deal with an especially aggressive firewall that filters ICMP, you can use a TCP-based technique by means of traceroute's -T (--tcp) option. This tries to address port 80 on the destination node using TCP SYN probes and recommends itself particularly well if the destination node is a web server.

You can request a different port by means of the -p ex_port (--port=ex_port) option. The TCP-based technique does not actually open a connection to the destination node and, therefore, stays invisible to application programs there.

Basically, the tracepath command does the same thing as traceroute, but does not offer most of its advanced options and can be invoked by regular users (some of traceroute's options can only be used by the root user). In addition, tracepath determines the maximum transmission unit (MTU).

Like traceroute, tracepath outputs the addresses of all routers on the route to the destination node. The remainder of the lines show the time the packets took, as well as additional information. For example, asymm 5 would mean that the router's answer took five hops instead of the four hops of the request, but this information is not always reliable.

Socket Tests

ss

A network socket (TCP/IP socket) is an internal endpoint for sending or receiving data within a node on a computer network. Specifically, it is a representation of this endpoint in networking software (the protocol stack).

A network socket example is an entry in a table consisting of the IP address and port number of a local node, together with the IP address and port number of a remote node. A TCP/IP socket is a form of system resource.

TCP/IP sockets are different from a UNIX domain socket (also known as an Inter-process Communication (IPC) socket), which is a data communications endpoint for exchanging data between processes executing on the same host operating system. Since these sockets know that they are executing on the same system, they can avoid some checks and operations (e.g., routing), which makes them faster and lighter than network sockets.

If you want to run a service, but client hosts cannot connect to it, being rejected with error messages like Unable to connect to remote host: Connection refused, you should ensure that the service actually listens for connections as it should. You can do this with the ss command.

By default, the ss (socket statistics) command shows open non-listening TCP, UDP, and UNIX domain sockets that have an established connection.

In ss's output, the following columnar data is provided:

Netid

The socket type and transport protocol. Possible values include:

• tcp

• udp

• raw

• u_str (unix_stream)

• u_dgr (UNIX datagram sockets)

• nl (netlink)

• p_raw (raw packet sockets)

• p_dgr (datagram packet sockets)

State

The socket state. The names are TCP states, except for UNCONN, which is normal for non-TCP unconnected sockets.

Recv-Q

Amount of data queued for receipt.

Send-Q

Amount of data queued for transmission.

Local Address:Port and Peer Address:Port

The local address/port of the socket and its peer address/port (if the socket is connected).

ss Options

Helpful ss options include:

-a, --all

Display both listening and non-listening (for TCP, this means established connections) sockets.

-l, --listening

Display only listening sockets (these are omitted by default).

-n, --numeric

Do not try to resolve service names.

-p, --processes

Show process using socket.

-s, --summary

Print summary statistics.

-t, --tcp

Display TCP sockets.

-u, --udp

Display UDP sockets.

-x, --unix

Display UNIX domain sockets.

ss Commands

ss -atu, ss --all --tcp --udp

Display listening and non-listening TCP and UDP sockets.

ss -ltu, ss --listening --tcp --udp

Display listening TCP and UDP sockets.

If a service does not show up in the output of ss -ltu, this indicates that the associated program is not running. If the service does occur in the list, but does not appear to be working, it is possible that clients are being rejected by a firewall before they reach it.

Also, it is possible that the port in question is blocked by another program, which, for some reason, does not work correctly. In this case, you can use ss -lptu to display the process ID (PID) and name of the program serving the port.

ss -ptu, ss --processes --tcp --udp

Display non-listening TCP and UDP sockets, along with the processes using the sockets.

lsof

The lsof (list open files) command is used to list open files. Several of its options are germane to GNU/Linux networking:

-i

Display a listing of all Internet and x.25 (HP-UX) network files.

The -i option can be passed Internet addresses to provide more specific output. Accepted values can be:

• The IP version (e.g., 4, 6)

• Protocol (TCP, UDP)

• Hostname

• Host address (an IP address in numerical form)

• Service (has to be a name present in /etc/services)

• Port number

-n

Disable name resolution.

-P

Inhibit the conversion of port number to port names for network files.

-s

Display file size at all times.

When combined with the p:s syntax, lets you specify a protocol and state you would like to target for the lsof output. TCP and UDP are accepted protocols.

Common TCP state names are:

• CLOSED

• IDLE

• BOUND

• LISTEN

• ESTABLISHED

• SYN_SENT

• SYN_RECEIVED

• ESTABLISHED

• CLOSE_WAIT

• FIN_WAIT1

• CLOSING

• LAST_ACK

• FIN_WAIT2

• TIME_WAIT

Two common UDP state names are Unbound and Idle.

lsof Commands

lsof -iTCP -sTCP:LISTEN

View TCP files with TCP state of LISTEN.

lsof -U

View list of UNIX domain socket files currently in use.

nmap

Commands like ss assume that you have shell access to a system. If you want to examine a system's ports from the outside, you can use the nmap (network mapper) command.

nmap is a port scanner that checks for open, filtered, and unused TCP and UDP ports on a computer over the network. The command target can also be a firewall infrastructure, therefore nmap can help you uncover gaps in your security strategy.

Likely, nmap is not available on your GNU/Linux distribution by default, so it will need to be installed:

# apt install nmap (Debian)

# dnf install nmap (Fedora)

Here is a scan report for scanme.nmap.org:

$ nmap scanme.nmap.org
Starting Nmap 7.70 ( https://nmap.org ) at 2021-07-31 08:43 PDT
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up (0.092s latency).
Other addresses for scanme.nmap.org (not scanned): 2600:3c01::f03c:91ff:fe18:bb2f
Not shown: 988 closed ports
PORT      STATE    SERVICE
22/tcp    open     ssh
25/tcp    filtered smtp
53/tcp    open     domain
80/tcp    open     http
139/tcp   filtered netbios-ssn
445/tcp   filtered microsoft-ds
513/tcp   open     login
514/tcp   open     shell
1900/tcp  filtered upnp
2869/tcp  filtered icslap
9929/tcp  open     nping-echo
31337/tcp open     Elite

Nmap done: 1 IP address (1 host up) scanned in 6.27 seconds

nmap considers ports open if a service can be reached. Ports for which the target host returns an error message are marked closed, while ports where there is no reaction at all (e.g., the inquiry packets are thrown away by the target host or a firewall, and not even an error message is sent in reply) are designated filtered.

By default, nmap analyzes the target host's TCP ports using an implicit SYN scan, i.e., for each of the ports under consideration, nmap sends a TCP segment with the SYN flag set (as if it wanted to start a new connection). If the target host answers with a TCP segment that has the SYN_ACK flag set, nmap takes no further action (specifically, it does not acknowledge the segment).

So, the half-open connection is thrown out by the target host after the statutory timeouts have occurred. If, instead, the target host answers with a segment with the RST flag set, the port is closed. If, after several tries there is no answer or only ICMP unreachable messages, the port is set to filtered.

SYN scans require root privileges. If you do not have root access (and therefore cannot send raw packets), nmap automatically does a TCP connect scan, which does not require special privileges. Other techniques that nmap offers include a UDP scan and several other variants of TCP-based scans, e.g., to discover firewall rule sets. Run man 1 nmap for more information.

nmap can not only identify the active ports on a host, but can, in many cases, tell you which software is used to serve the ports. For this, you need to specify the -A option. To do this, nmap relies on a database of signatures of diverse programs that come with the software.

You should run nmap both from a different system on your LAN, and from a different system on a different network against the same host. This will tell you what ports are open on your system and which services are allowed through your host’s firewall.

nmap Commands

nmap -sn ex_network_address/ex_netmask

Scan network to determine client IP addresses. -sn is used to initiate a ping scan.

Some firewalls or hosts will not respond to PING, so you may need to use the -Pn parameter with your command (e.g., nmap -Pn -sn) to tell nmap to skip host discovery. Instead of only returning responses for hosts that responded, this returns an entry for every valid address in the network.

nmap ex_network_address/ex_netmask

Determine services running on a network's systems. The default scan here scans 1000 TCP ports. Host discovery will take place.

nmap ex_ip_address_start-end

Determine services running on hosts over a range of IP addresses (e.g., nmap 192.168.1.1-20). The default scan here scans 1000 TCP ports. Host discovery will take place.

nmap ex_target

Determine services running on a single host. ex_target can be a hostname or IP address. The default scan scans 1000 TCP ports and host discovery takes place.

nmap -iL ex_file.txt

Determine services running on hosts listed in a text file.

# nmap -O ex_target

Extract as much information as possible about a host. -O is for os detection.

# nmap -A ex_target

Get more information about the services running on a host. -A means enable OS detection, version detection, script scanning, and traceroute.

# nmap -sS ex_target

Scan a host using TCP SYN. -sS means TCP SYN/Connect()/ACK/Window/Maimon scans.

nmap -sT ex_target

Scan a host's TCP ports. -sT means TCP connect scan.

# nmap -sU ex_target

Scan a host's UDP ports. -sU means UDP scan.

nmap -p ex_port ex_target

Scan a single port on a host. -p means only scan specified ports.

nmap -p ex_start_port-end_port ex_target

Scan a range of ports on a host.

nmap -F ex_target

Scan 100 most common ports on a host. -F means Fast mode.

nmap -p- ex_target

Scan all 65,535 ports on a host. -p- means to scan ports from 1 through 65,535.

netcat

The netcat command is used for arbitrary socket connections, listens, concatenations, and redirects. On Debian, the command nc is used for netcat, while on Fedora, ncat is used instead.

From the nc manual:

The nc (or netcat) utility is used for just about anything under the sun involving TCP, UDP, or UNIX-domain sockets. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6. Unlike telnet(1), nc scripts nicely, and separates error messages onto standard error instead of sending them to standard output, as telnet(1) does with some.

From the ncat manual:

Ncat is a feature-packed networking utility which reads and writes data across networks from the command line. Ncat was written for the Nmap Project and is the culmination of the currently splintered family of Netcat incarnations. It is designed to be a reliable back-end tool to instantly provide network connectivity to other applications and users. Ncat will not only work with IPv4 and IPv6 but provides the user with a virtually limitless number of potential uses.

Among Ncat's vast number of features there is the ability to chain Ncats together; redirection of TCP, UDP, and SCTP ports to other sites; SSL support; and proxy connections via SOCKS4, SOCKS5 or HTTP proxies (with optional proxy authentication as well). Some general principles apply to most applications and thus give you the capability of instantly adding networking support to software that would normally never support it.

The basic syntax for the netcat command is:

netcat ex_options ex_host ex_port

ex_host can be a hostname or IP address. By default, netcat will try to start a TCP connection, but if you want a UDP connection, you can use the -u option. Press Ctrl+c to close a connection.

netcat can be used for a variety of purposes.

Create a Chat or Web Server

First, start netcat listening on a port number on the first host. Choose a port that is not already in use by another application or service (e.g., 56938).

netcat -l -p ex_port

Then, on the second host, launch a chat session with a new TCP connection on the port that was previously specified to connect to as the listening port on the first host (i.e., 56938).

netcat ex_host ex_port

Now, when you enter a message on the terminal, it will be shown on both hosts.

The same technique can be used to start a basic local web server. First, create an index.html file in your system's local file system. Then, run the following command from the same directory that you saved your index.html file to (again, choosing a port that is not already in use by another application or service, e.g., 60948):

printf 'HTTP/1.1 200 OK\n\n%s' "$(cat 'index.html')" | netcat -l ex_port

In a browser, navigate to localhost:ex_port to see the contents of your index.html file rendered in the browser.

Sending Files

netcat can be used to send files from one host to another by creating a basic client/server model. The receiving host listens on a port that you specify, and the sending host establishes a TCP connection to the receiving host on that port to send the file over.

On the receiving host, run the following command to open a port (e.g., 50982) for an incoming connection and redirect the output to a file (i.e., the file to be received):

netcat -l ex_port > ex_filename

On the sending host, connect to the receiving host and send the file:

netcat ex_host ex_port < ex_file

To transfer a directory, you can use the tar command.

On the receiving host, run:

netcat -l ex_port | tar -xzf -

On the sending host, run:

tar -czf - ex_directory | netcat ex_host ex_port

The transfer progress will be displayed on both ends of the connection.

Port Scanning

netcat can be used as a simpler alternative to nmap for port scanning. For example, the following command will determine if the ex_port TCP port is open, without sending any data to it:

netcat -z -v ex_host ex_port

The -z option specifies Zero-I/O mode and -v causes netcat to produce more verbose output.

ex_port can be a range, as well (e.g., 30-90).

To scan for UDP ports, add the -u option.

Name Resolution Tests

If connections to hosts addressed by name take too long to set up or fail to be established after a delay, but trying to make the same connection based on the IP address is fast, the Domain Name System (DNS) may be to blame. Conversely, your system may take a long time to connect because the remote host tries to find a name for your IP address and runs into some problem on their side.

To test DNS, you can use the host and dig commands.

$ host duckduckgo.com
duckduckgo.com has address 107.20.240.232
duckduckgo.com mail is handled by 0 duckduckgo-com.mail.protection.outlook.com.

host can accept an IP address as an argument too:

$ host 107.20.240.232
232.240.20.107.in-addr.arpa domain name pointer ec2-107-20-240-232.compute-1.amazonaws.com.

You can compare the output of several DNS servers by specifying their IP addresses (or hostnames) as part of your query:

$ host duckduckgo.com 192.168.2.3
Using domain server:
Name: 192.168.2.3
Address: 192.168.2.3#53
Aliases:

duckduckgo.com has address 107.20.240.232
duckduckgo.com mail is handled by 0 duckduckgo-com.mail.protection.outlook.com.

This way, you can check whether a DNS server gives the correct answers.

You can request specific types of DNS records by using host's -t ex_type option:

$ host -t mx duckduckgo.com
duckduckgo.com mail is handled by 0 duckduckgo-com.mail.protection.outlook.com.

With the -l option, you can have host perform a zone transfer and print out the NS, PTR, and A/AAAA records for the zone name (if you have the appropriate permissions). If you add the -a option to -l, you can print all the DNS records in the zone.

Essentially, the dig command does what the host command does, but allows for detailed analysis by providing more comprehensive output.

$ dig duckduckgo.com

; <<>> DiG 9.11.14-RedHat-9.11.14-2.fc31 <<>> duckduckgo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65481
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 70129b5efe1f841ce629fade5ebf144fa5296334dfd9f7ee (good)
;; QUESTION SECTION:
;duckduckgo.com.            IN  A

;; ANSWER SECTION:
duckduckgo.com.     200 IN  A   184.72.104.138

;; AUTHORITY SECTION:
duckduckgo.com.     51381   IN  NS  dns1.p05.nsone.net.
duckduckgo.com.     51381   IN  NS  dns3.p05.nsone.net.
duckduckgo.com.     51381   IN  NS  dns2.p05.nsone.net.
duckduckgo.com.     51381   IN  NS  dns4.p05.nsone.net.

;; Query time: 65 msec
;; SERVER: 10.0.2.3#53(10.0.2.3)
;; WHEN: Fri May 15 15:14:39 PDT 2020
;; MSG SIZE  rcvd: 176

To resolve IP addresses into hostnames, use dig's -x ex_ip_address option:

$ dig -x 184.72.104.138

; <<>> DiG 9.11.14-RedHat-9.11.14-2.fc31 <<>> -x 184.72.104.138
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19821
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 2dc48dada266079651fc81035ebf186a52ce8883004eb65d (good)
;; QUESTION SECTION:
;138.104.72.184.in-addr.arpa.   IN  PTR

;; ANSWER SECTION:
138.104.72.184.in-addr.arpa. 300 IN PTR ec2-184-72-104-138.compute-1.amazonaws.com.

;; AUTHORITY SECTION:
72.184.in-addr.arpa.    86400   IN  NS  pdns1.ultradns.net.
72.184.in-addr.arpa.    86400   IN  NS  pdns2.ultradns.net.
72.184.in-addr.arpa.    86400   IN  NS  pdns3.ultradns.org.

;; Query time: 176 msec
;; SERVER: 10.0.2.3#53(10.0.2.3)
;; WHEN: Fri May 15 15:32:10 PDT 2020
;; MSG SIZE  rcvd: 224

To use a specific name server with dig, use the @ex_name_server syntax, e.g., dig @192.168.2.3 duckduckgo.com.

Packet Analysis

Packet analysis is the interception and logging of traffic that passes over a network or a part of a network. A popular packet analyzer is tcpdump.

tcpdump

tcpdump analyzes packets moving through a network interface. It switches a network adapter into promiscuous mode, causing it to read and report all packets (not just those only addressed to the local interface, as it normally does). tcmpdump can only be used by the root user.

Likely, you will need to install tcpdump from your distribution's repository:

# apt install tcpdump (Debian)

# dnf install tcpdump (Fedora)

Above, the -i ex_interface (--interface=ex_interface) option is used to determine with network interface will be monitored (e.g., bridge0). The -n option can be added to disable DNS lookups and port name resolution.

The -w ex_file option can be used to write tcpdump's output to a file, instead of the standard output:

# tcpdump -i ex_interface -w ex_file.cap

The data in this file can be viewed by using the -r ex_file option:

# tcpdump -r ex_file.cap

A graphical packet analyzer alternative to tcpdump is Wireshark, which comes with an extensive feature set. It is a GUI program that allows for detailed analysis of all network packets.

wireshark's output consists of three window panes:

1. The topmost pane displays incoming packets.
2. The bottom-most pane decodes the data in hexadecimal notation.
3. The center pane allows for the convenient and detailed dissection of header/payload information.

Like tcpdump, wireshark is not a standard GNU/Linux tool and usually needs to be specifically installed.

Additional Networking Commands

ethtool

ethtool can be used to query or control network driver and hardware settings, particularly for wired Ethernet devices. On Debian systems, you will need to install this command (# apt install ethtool).

For example, the ethtool ex_device_name command can be used to print current settings for a network interface.

wget

The wget command is a non-interactive downloader of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.

For example, the following command downloads the index page of lpi.org to the current directory:

$ wget 'https://www.lpi.org/'
--2021-07-31 09:37:01--  https://www.lpi.org/
Resolving www.lpi.org (www.lpi.org)... 65.39.134.146
Connecting to www.lpi.org (www.lpi.org)|65.39.134.146|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                     [  <=>                                 ]  62.13K   219KB/s    in 0.3s    

2021-07-31 09:37:02 (219 KB/s) - ‘index.html’ saved [63624]

To change the name of the downloaded file, or to specify which directory to save the file to, you can use the -O ex_file (--output-document=ex_file) option:

$ wget -O 'Downloads/lpi.txt' 'https://www.lpi.org/'
--2021-07-31 09:38:24--  https://www.lpi.org/
Resolving www.lpi.org (www.lpi.org)... 65.39.134.146
Connecting to www.lpi.org (www.lpi.org)|65.39.134.146|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘Downloads/lpi.txt’

Downloads/lpi.txt              [  <=>                                 ]  62.13K   222KB/s    in 0.3s    

2021-07-31 09:38:25 (222 KB/s) - ‘Downloads/lpi.txt’ saved [63624]

curl

curl is a command to transfer data from or to a server. It supports the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP. On Debian systems, you will need to install this command (# apt install curl).

For example, you can use curl and its -I (--head) option to fetch the header for a document:

$ curl -I 'https://www.lpi.org/'
HTTP/1.1 200 OK
Date: 2021-09-19 18:08
Server: Apache
X-Content-Type-Options: nosniff
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: no-cache, must-revalidate
X-Content-Type-Options: nosniff
Content-Language: en
X-Frame-Options: sameorigin
Permissions-Policy: interest-cohort=()
X-UA-Compatible: IE=edge
Content-Type: text/html; charset=utf-8

The URL syntax for curl is protocol-independent and there are many uses of curl.

Some example commands include:

curl ex_url...

Download a URL. By default, curl sends its output to the standard output.

curl -O ex_url..., curl --remote-name ex_url...

Download a URL and save it as a file to the current directory. The name of the file will be the name of the ex_url file. This will only work if the URL provided ends with a filename.

curl -o ex_directory/ex_filename ex_url, curl --output ex_directory/ex_filename ex_url

Download a URL and save it as a file in the specified directory with the specified filename.

curl -C - -O ex_url..., curl --continue-at - --remote-name ex_url...

Resume a download that has been previously interrupted. The - offset provided to the -C option tells curl to automatically find out where/how to resume the transfer.

curl -u ex_username:ex_password ex_ftp_url, curl --user ex_username:ex_password ex_ftp_url

Download a URL and save it as a file from a user authenticated FTP server.

curl -u ex_username:ex_password -T ex_file ex_ftp_url, curl --user ex_username:ex_password --upload-file ex_file ex_ftp_url

Upload a local file to a user authenticated FTP server.

curl -x ex_proxy_name:ex_proxy_port ex_url, curl --proxy ex_proxy_name:ex_proxy_port ex_url

Use a proxy to access a URL.

curl -u ex_username:ex_password -x ex_proxy_name:ex_proxy_port ex_url, curl --user ex_username:ex_password --proxy ex_proxy_name:ex_proxy_port ex_url

Use a proxy that requires authentication to access a URL.

If you want to disable curl's progress meter, use it with the -s (--silent) option.

ftp

The ftp command is used to handle file transfers for servers on a trusted local network. By default, ftp is not secure, so it is usually best to avoid it for transferring files over the Internet.

To connect to a server:

ftp ex_host

ex_host can be a hostname or IP address.

If your local machine's account username matches that of the FTP user on the server, you can press Enter to continue. If the FTP user on the server is distinct, you will need to enter the FTP username to proceed.

Next, you will need to enter the password for the FTP user account. If you successfully authenticate, you will be brought to the ftp> prompt.

Useful ftp commands include:

ls

Print a listing of the contents of the current directory on the remote machine. This command can be provided a directory path as an argument to list the contents of that specific directory.

cd ex_remote_directory

Change the working directory on the remote machine to ex_remote_directory.

get ex_remote_file

Retrieve ex_remote_file and store it on the local machine.

put ex_local_file

Store ex_local_file on the remote machine.

cdup

Change the remote machine working directory to the parent of the current remote machine working directory.

rename ex_old_name ex_new_name

Rename remote machine file from ex_old_name to ex_new_name.

delete ex_remote_file

Delete ex_remote_file on the remote machine.

close

Terminate the FTP session with the remote server and return to the command interpreter.

sftp

sftp is a secure file transfer program that is similar to ftp. Unlike ftp, it performs all operations over an encrypted Secure Shell (SSH) transport. Also, it uses many features of SSH, e.g., public key authentication and compression.

To connect to a server:

sftp ex_user@ex_host

ex_host can be a hostname or IP address.

The connection process for sftp is identical to the ssh connection process.

Useful sftp commands include:

ls

Print a listing of the contents of the current directory on the remote machine. This command can be provided a directory path as an argument to list the contents of that specific directory.

cd ex_remote_directory

Change the working directory on the remote machine to ex_remote_directory.

get ex_remote_file

Retrieve ex_remote_file and store it on the local machine.

put ex_local_file

Store ex_local_file on the remote machine.

cd ..

Change the remote machine working directory to the parent of the current remote machine working directory.

rename ex_old_name ex_new_name

Rename remote machine file from ex_old_name to ex_new_name.

rm ex_remote_file

Delete ex_remote_file on the remote machine.

exit

Terminate the SFTP session with the remote server and return to the command interpreter.

Documentation

You can find more information on the commands discussed above by examining the Linux User's Manual, either at the command line or online.