Implementing a Tail-like Function

Here’s a concise roadmap of three approaches to implement a tail‑like function in Python.

• Full‑file read: Read the entire file into memory and slice the last n lines; simple but uses O(file_size) memory.
• Fixed‑size queue: Stream lines one by one, retaining only the last n in a collections.deque(maxlen=n), so memory is bounded by O(n).
• Backward block buffer: Seek from the file’s end in binary mode, read fixed‑size blocks backward, count newlines until enough are found, then decode and print - uses minimal memory even for very large files.

1. Naïve Full‑File Read

Code Example

1
def tail_simple(num_lines: int, file_path: str = "./file.log") -> None:
2
    with open(file_path, "r", encoding="utf-8", errors="replace") as f:
3

4
        lines = f.readlines()
5

6
        for line in lines[-num_lines:]:
7
            print(line.rstrip("\n"))

Nuances

• Memory usage: f.readlines() loads every line into a list, so memory scales with file size.
• Performance: Fast for small to medium files, but will crash or swap if the file is huge.

2. Bounded Memory via a Queue

Code Example

1
from collections import deque
2

3
def tail_queue(num_lines: int, file_path: str = "./file.log") -> None:
4

5
    buffer = deque(maxlen=num_lines)
6
    with open(file_path, "r", encoding="utf-8", errors="replace") as f:
7
        for line in f:
8

9
            buffer.append(line.rstrip("\n"))
10
    for line in buffer:
11
        print(line)

Nuances

• Memory bound: Uses at most n lines of memory, regardless of file size.
• Streaming: Reads line by line; useful for very large files or when you don’t know file size in advance.
• Overhead: Python’s generator overhead per line; slightly slower than block reads but generally fine.

3. Block‑Buffered Reverse Read

When num_lines is small relative to file size, reading backwards in chunks can be far more efficient.

1
import os
2

3
def tail(num_lines: int, file_path: str = "./file.log") -> None:
4

5

6
    with open(file_path, "rb") as f:
7
        f.seek(0, os.SEEK_END)
8
        file_size = f.tell()
9
        buffer = bytearray()
10
        lines_found = 0
11
        block_size = 1024
12

13

14
        while file_size > 0 and lines_found <= num_lines:
15
            read_size = min(block_size, file_size)
16
            f.seek(file_size - read_size)
17
            data = f.read(read_size)
18
            buffer[:0] = data
19
            lines_found = buffer.count(b"\n")
20
            file_size -= read_size
21

22

23
        lines = buffer.splitlines()[-num_lines:]
24
        for line in lines:
25
            print(line.decode("utf-8"))

Nuances

• Binary mode avoids issues with variable‑length encodings when seeking.
• Block size: Choose based on typical line length; too small and you incur many seeks, too large and you waste memory.
• Byte vs. text splitting: Counts b"\n" and splits on raw bytes, then decodes.
• Edge cases: Handles files smaller than block_size, and files with fewer than num_lines lines gracefully.

Conclusion

• Full‑file read is easiest to implement but not scalable.
• Deque‑based offers O(n) memory bound and simple streaming.
• Block‑buffered reverse read provides near‑constant memory overhead and is ideal for very large files.

Choose the approach that best fits your file sizes and memory constraints.