Deploying a TCP Server to Fly.io: A Technical Report

· nat's blog

Overview #

This document describes the process of creating and deploying a minimal C-based TCP server to Fly.io. The server accepts connections on port 1848 and responds with "hello\n" before closing the connection. Running a service on TCP can be more costly though, than relying on HTTP or HHTPS.

Project Structure #

fly-test/
├── main.c           # Server implementation
├── CMakeLists.txt   # Build configuration
├── Dockerfile       # Container image definition
└── fly.toml         # Fly.io deployment configuration

Implementation #

Server Code #

The server is implemented in C using standard POSIX socket APIs. The i mplementation uses an IPv6 socket configured to accept both IPv4 and IPv6 connections.

 1#include <stdio.h>
 2#include <stdlib.h>
 3#include <string.h>
 4#include <unistd.h>
 5#include <sys/socket.h>
 6#include <netinet/in.h>
 7
 8int main(int argc, char *argv[])
 9{
10   int server_fd, client_fd;
11   struct sockaddr_in6 address;
12   int opt = 1;
13   int addrlen = sizeof(address);
14   int port = 1848;
15
16   if (argc > 1)
17      port = atoi(argv[1]);
18
19   // Create IPv6 socket
20   if ((server_fd = socket(AF_INET6, SOCK_STREAM, 0)) == 0)
21   {
22      perror("socket failed");
23      exit(EXIT_FAILURE);
24   }
25
26   // Enable address reuse
27   if (setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)))
28   {
29      perror("setsockopt");
30      exit(EXIT_FAILURE);
31   }
32
33   // Disable IPv6-only mode to accept IPv4 connections
34   int no = 0;
35   if (setsockopt(server_fd, IPPROTO_IPV6, IPV6_V6ONLY, &no, sizeof(no)))
36   {
37      perror("setsockopt IPV6_V6ONLY");
38      exit(EXIT_FAILURE);
39   }
40
41   // Bind to all interfaces
42   address.sin6_family = AF_INET6;
43   address.sin6_addr = in6addr_any;
44   address.sin6_port = htons(port);
45
46   if (bind(server_fd, (struct sockaddr *)&address, sizeof(address)) < 0)
47   {
48      perror("bind failed");
49      exit(EXIT_FAILURE);
50   }
51
52   if (listen(server_fd, 3) < 0)
53   {
54      perror("listen");
55      exit(EXIT_FAILURE);
56   }
57
58   printf("Server listening on port %d\n", port);
59
60   // Accept connections and respond
61   while (1)
62   {
63      if ((client_fd = accept(server_fd, (struct sockaddr *)&address, 
64                              (socklen_t*)&addrlen)) < 0)
65      {
66         perror("accept");
67         continue;
68      }
69
70      write(client_fd, "hello\n", 6);
71      close(client_fd);
72   }
73
74   close(server_fd);
75   return 0;
76}

The critical aspect is the use of AF_INET6 with IPV6_V6ONLY disabled. This configuration allows the socket to accept connections on both IPv4 and IPv6 addresses.

Build Configuration #

CMakeLists.txt provides a minimal build configuration:

1cmake_minimum_required(VERSION 3.10)
2project(hello-server C)
3add_executable(hello-server main.c)

Container Image #

The Dockerfile uses a multi-stage build to separate compilation from runtime:

1FROM --platform=linux/amd64 alpine:latest AS builder
2RUN apk add --no-cache gcc musl-dev cmake make
3WORKDIR /build
4COPY main.c CMakeLists.txt ./
5RUN cmake . && make
6
7FROM --platform=linux/amd64 alpine:latest
8COPY --from=builder /build/hello-server /app/hello-server
9CMD ["/app/hello-server"]

The builder stage installs compilation tools and builds the binary. The runtime stage contains only the Alpine base image and the compiled binary. This approach keeps the source code out of the production image and reduces the final image size to approximately 91 MB.

The platform is explicitly set to linux/amd64 to match Fly.io's infrastructure.

Pain Point #

This could be a pain point in the future. Automatic discovery of the fly.io hardware would be preferable.

Fly.io Configuration #

Service Definition #

The fly.toml file defines a TCP service rather than an HTTP service:

 1app = "hello-server"
 2primary_region = "ams"
 3
 4[build]
 5
 6[[services]]
 7  internal_port = 1848
 8  protocol = "tcp"
 9
10  [[services.ports]]
11    port = 1848
12    handlers = []
13
14[[vm]]
15  cpu_kind = "shared"
16  cpus = 1
17  memory_mb = 256

The [[services]] section specifies:

The VM configuration allocates minimal resources: 1 shared CPU and 256 MB of memory.

Deployment Process #

1# Create the application
2flyctl apps create hello-server
3
4# Deploy the application
5flyctl deploy
6
7# Verify deployment
8flyctl status

Fly.io allocated two IP addresses:

The deployment created two machines in the Amsterdam region for high availability.

Network Architecture #

The server binds to :::1848, which represents the IPv6 wildcard address. With IPV6_V6ONLY disabled, this binding accepts connections on both IPv4-mapped addresses and native IPv6 addresses.

Verification on the deployed machine:

1$ flyctl ssh console -C "netstat -tlnp | grep 1848"
2tcp  0  0  :::1848  :::*  LISTEN  641/hello-server

The :::1848 notation confirms IPv6 wildcard binding with IPv4 compatibility.

Testing #

Local Testing #

1$ cd build && ./hello-server &
2$ echo | nc localhost 1848
3hello
4$ echo | nc ::1 1848
5hello

Both IPv4 (localhost) and IPv6 (::1) connections work correctly locally.

Remote Testing #

From a server with a routable IPv6 address:

1$ telnet -6 hello-server.fly.dev 1848
2Trying 2a09:8280:1::c5:a273:0...
3Connected to hello-server.fly.dev.
4Escape character is '^]'.
5hello
6Connection closed by foreign host.

IPv6 connections function reliably.

IPv4 connections show different behavior:

1$ telnet -4 hello-server.fly.dev 1848
2Trying 66.241.125.63...
3Connected to hello-server.fly.dev.
4Escape character is '^]'.
5Connection closed by foreign host.

The connection establishes but the response is not received. This behavior depends on the client's network configuration and IPv6 capabilities.

Why it fails #

Of course for many ipv6.

Check your setup with test-ipv6.com.

It seems you will likely need to get an IPv4 address (at cost).

1fly ips allocate-v4

This will up the cost of the whole installation from nominally $2 when doing nothing to $26. Ouch.

Issues and Considerations #

IPv4-Only Socket Binding #

Initial implementation used AF_INET with INADDR_ANY, resulting in binding to 0.0.0.0:1848. This configuration only accepts IPv4 connections. On Fly.io, where IPv6 is the primary protocol, connections would establish but immediately close without data transfer.

The solution requires using AF_INET6 with in6addr_any and explicitly disabling IPV6_V6ONLY. This creates a dual-stack socket that accepts both protocol versions.

Service Type Configuration #

Using [http_service] in fly.toml causes Fly.io to expect HTTP protocol traffic. The proxy attempts to parse HTTP requests and responses, which fails with raw TCP data. The correct configuration uses [[services]] with protocol = "tcp" and an empty handlers array.

IPv4 Connectivity Limitations #

Fly.io's IPv4 addresses are shared and routing depends on the client's network configuration. Clients without proper IPv6 support or those behind certain NAT configurations may experience issues with IPv4 connections. IPv6 connections are more reliable on Fly.io's infrastructure.

Testing from virtual machines or Docker containers without routable IPv6 addresses may show inconsistent behavior. Production deployments should account for this by ensuring clients can connect via IPv6 or by implementing additional routing logic.

Port Selection (Voodoo) #

Standard ports like 80, 443, or 8080 may have special handling in Fly.io's infrastructure. Using a non-standard port (1848 in this case) avoids potential conflicts with HTTP-specific routing rules.

Debugging Techniques #

The flyctl ssh console command provides direct access to the running machine. Useful commands for debugging:

1# Check listening ports
2netstat -tlnp | grep <port>
3
4# View running processes
5ps aux | grep <process>
6
7# Test local connectivity
8nc localhost <port>

The netstat output format differs between IPv4 and IPv6:

Resource Allocation #

The minimal configuration (256 MB RAM, 1 shared CPU) is sufficient for a simple TCP server. Fly.io's auto-start and auto-stop features allow scaling to zero when idle, reducing costs for low-traffic applications.

Heartlessly Vibecoded

last updated: